{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Mathematical Engineering of Deep Learning\n",
    "\n",
    "## Practical 5 (Julia version)\n",
    "**For an R or Python version see the [course website](https://deeplearningmath.org/)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this practical we deal with a reduced version of the CIFAR10 dataset and train convolutional neural networks. We use a small subset of the dataset to be able to compute things in sensbile time within the tutorial.\n",
    "\n",
    "The main focus is to see how \"to assemble\" different convolutional layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CIFAR10\n",
    "See [CIFAR10 website](https://www.cs.toronto.edu/~kriz/cifar.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size(CIFAR10Fulltrain_x) = (32, 32, 3, 50000)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "10-element Array{String,1}:\n",
       " \"airplane\"\n",
       " \"automobile\"\n",
       " \"bird\"\n",
       " \"cat\"\n",
       " \"deer\"\n",
       " \"dog\"\n",
       " \"frog\"\n",
       " \"horse\"\n",
       " \"ship\"\n",
       " \"truck\""
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using MLDatasets\n",
    "CIFAR10Fulltrain_x, CIFAR10Fulltrain_y = CIFAR10.traindata()\n",
    "CIFAR10Fulltest_x,  CIFAR10Fulltest_y  = CIFAR10.testdata()\n",
    "@show size(CIFAR10Fulltrain_x)\n",
    "classNames = CIFAR10.classnames()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We consider only 3 classes - not all 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "false"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "in(7,[2,34,5,2])\n",
    "7 ∈ [2,34,5,2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15000.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "50000*3/10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(trainFilter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length(CIFAR10Filteredtrain_y) = 15000\n",
      "length(CIFAR10Filteredtest_y) = 3000\n"
     ]
    }
   ],
   "source": [
    "#use only \"airplane\", \"cat\" and \"truck\" to make the problem \"easier and quicker\" for the practical\n",
    "usedLabels = [0,3,9]\n",
    "trainFilter = [y ∈ usedLabels for y in CIFAR10Fulltrain_y]  #use \\in +[TAB] to make ∈ (element of set checking)\n",
    "testFilter = [y ∈ usedLabels for y in CIFAR10Fulltest_y]\n",
    "CIFAR10Filteredtrain_x, CIFAR10Filteredtrain_y = CIFAR10Fulltrain_x[:,:,:,trainFilter], CIFAR10Fulltrain_y[trainFilter]\n",
    "CIFAR10Filteredtest_x,  CIFAR10Filteredtest_y  = CIFAR10Fulltest_x[:,:,:,testFilter],  CIFAR10Fulltest_y[testFilter];\n",
    "@show length(CIFAR10Filteredtrain_y)\n",
    "@show length(CIFAR10Filteredtest_y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We stay with a small training data size and validation data size - also due to time constraints. For testing we could use the full test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainLength = 2000\n",
    "validateLength = 1000;\n",
    "trainRange = 1:trainLength\n",
    "validateRange = (trainLength+1):(trainLength+validateLength);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Int64,1}:\n",
       " 0\n",
       " 3\n",
       " 9"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "usedLabels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creates the training, validation, and test sets \n",
    "using Flux: onehotbatch\n",
    "\n",
    "x_train = CIFAR10Filteredtrain_x[:,:,:,trainRange]\n",
    "x_validate = CIFAR10Filteredtrain_x[:,:,:,validateRange]\n",
    "x_test = CIFAR10Filteredtest_x\n",
    "\n",
    "y_train = onehotbatch(CIFAR10Filteredtrain_y[trainRange], usedLabels)\n",
    "y_validate = onehotbatch(CIFAR10Filteredtrain_y[validateRange], usedLabels)\n",
    "y_test = onehotbatch(CIFAR10Filteredtest_y, usedLabels);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3×2000 Flux.OneHotMatrix{Array{Flux.OneHotVector,1}}:\n",
       " 0  0  0  0  0  0  0  0  0  1  1  0  0  …  1  0  1  0  0  0  1  0  0  1  0  0\n",
       " 0  0  1  0  0  0  1  1  1  0  0  0  1     0  0  0  1  1  0  0  1  1  0  0  0\n",
       " 1  1  0  1  1  1  0  0  0  0  0  1  0     0  1  0  0  0  1  0  0  0  0  1  1"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "40-element Array{UnitRange{Int64},1}:\n",
       " 1:50\n",
       " 51:100\n",
       " 101:150\n",
       " 151:200\n",
       " 201:250\n",
       " 251:300\n",
       " 301:350\n",
       " 351:400\n",
       " 401:450\n",
       " 451:500\n",
       " 501:550\n",
       " 551:600\n",
       " 601:650\n",
       " ⋮\n",
       " 1401:1450\n",
       " 1451:1500\n",
       " 1501:1550\n",
       " 1551:1600\n",
       " 1601:1650\n",
       " 1651:1700\n",
       " 1701:1750\n",
       " 1751:1800\n",
       " 1801:1850\n",
       " 1851:1900\n",
       " 1901:1950\n",
       " 1951:2000"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batchSize = 50\n",
    "[r for r in Iterators.partition(trainRange, batchSize)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "40"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Setup training mini-batches\n",
    "using Flux: onehotbatch\n",
    "batchSize = 50\n",
    "\n",
    "x_train_batches = [x_train[:, :, :, r] for r in Iterators.partition(trainRange, batchSize)];\n",
    "y_train_batches = [y_train[:,r] for r in Iterators.partition(trainRange,batchSize)];\n",
    "\n",
    "numMiniBatches = length(x_train_batches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32×32×3×50 Array{N0f8,4} with eltype FixedPointNumbers.Normed{UInt8,8}:\n",
       "[:, :, 1, 1] =\n",
       " 0.604  0.549  0.549  0.533  0.506  …  0.682  0.643  0.686  0.647  0.639\n",
       " 0.494  0.569  0.545  0.537  0.553     0.553  0.553  0.612  0.612  0.62\n",
       " 0.412  0.49   0.451  0.478  0.533     0.173  0.451  0.604  0.624  0.639\n",
       " 0.4    0.486  0.576  0.518  0.729     0.169  0.443  0.576  0.514  0.569\n",
       " 0.49   0.588  0.541  0.592  0.843     0.224  0.455  0.608  0.369  0.169\n",
       " 0.608  0.596  0.518  0.71   0.792  …  0.204  0.447  0.631  0.4    0.075\n",
       " 0.675  0.682  0.667  0.796  0.643     0.173  0.443  0.627  0.424  0.078\n",
       " 0.706  0.698  0.698  0.816  0.588     0.188  0.455  0.655  0.502  0.29\n",
       " 0.557  0.525  0.671  0.816  0.541     0.31   0.486  0.647  0.604  0.525\n",
       " 0.435  0.431  0.753  0.796  0.467     0.678  0.584  0.596  0.612  0.467\n",
       " 0.416  0.522  0.859  0.702  0.369  …  0.851  0.627  0.639  0.714  0.431\n",
       " 0.427  0.639  0.918  0.663  0.424     0.651  0.557  0.643  0.702  0.388\n",
       " 0.482  0.753  0.898  0.643  0.424     0.38   0.384  0.522  0.49   0.239\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.455  0.557  0.694  0.718  0.522  …  0.31   0.271  0.243  0.137  0.024\n",
       " 0.4    0.376  0.396  0.475  0.49      0.286  0.278  0.2    0.082  0.039\n",
       " 0.373  0.388  0.396  0.357  0.4       0.282  0.271  0.173  0.055  0.098\n",
       " 0.353  0.373  0.345  0.369  0.384     0.329  0.294  0.153  0.043  0.2\n",
       " 0.282  0.349  0.404  0.357  0.345     0.341  0.31   0.169  0.055  0.267\n",
       " 0.235  0.314  0.369  0.302  0.314  …  0.369  0.463  0.4    0.231  0.353\n",
       " 0.22   0.255  0.255  0.271  0.353     0.282  0.337  0.216  0.192  0.455\n",
       " 0.302  0.329  0.325  0.38   0.369     0.169  0.333  0.125  0.212  0.525\n",
       " 0.369  0.361  0.353  0.345  0.271     0.133  0.122  0.09   0.318  0.549\n",
       " 0.357  0.376  0.31   0.298  0.278     0.404  0.302  0.165  0.404  0.561\n",
       " 0.341  0.302  0.267  0.251  0.267  …  0.373  0.267  0.239  0.482  0.561\n",
       " 0.31   0.278  0.263  0.278  0.341     0.486  0.369  0.365  0.514  0.561\n",
       "\n",
       "[:, :, 2, 1] =\n",
       " 0.694  0.627  0.608  0.576  0.537  …  0.643  0.608  0.655  0.604  0.58\n",
       " 0.537  0.6    0.573  0.557  0.573     0.537  0.545  0.604  0.596  0.58\n",
       " 0.408  0.49   0.451  0.475  0.545     0.18   0.467  0.627  0.631  0.612\n",
       " 0.396  0.506  0.6    0.522  0.729     0.161  0.443  0.6    0.51   0.529\n",
       " 0.514  0.631  0.588  0.616  0.863     0.2    0.424  0.6    0.345  0.125\n",
       " 0.651  0.643  0.569  0.757  0.847  …  0.18   0.412  0.604  0.361  0.035\n",
       " 0.745  0.737  0.722  0.871  0.737     0.145  0.424  0.639  0.42   0.055\n",
       " 0.78   0.741  0.741  0.89   0.698     0.145  0.439  0.678  0.506  0.267\n",
       " 0.612  0.545  0.69   0.875  0.635     0.271  0.482  0.647  0.584  0.49\n",
       " 0.471  0.435  0.765  0.859  0.573     0.675  0.612  0.596  0.592  0.431\n",
       " 0.42   0.498  0.855  0.761  0.478  …  0.792  0.612  0.635  0.694  0.396\n",
       " 0.408  0.612  0.914  0.722  0.537     0.537  0.502  0.639  0.686  0.365\n",
       " 0.475  0.753  0.929  0.729  0.545     0.361  0.392  0.541  0.506  0.247\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.459  0.561  0.694  0.718  0.525  …  0.314  0.278  0.251  0.145  0.024\n",
       " 0.396  0.38   0.4    0.482  0.502     0.29   0.29   0.208  0.086  0.035\n",
       " 0.373  0.396  0.404  0.369  0.42      0.275  0.278  0.169  0.047  0.086\n",
       " 0.349  0.376  0.349  0.38   0.408     0.306  0.298  0.141  0.024  0.176\n",
       " 0.275  0.349  0.404  0.369  0.361     0.333  0.322  0.173  0.051  0.251\n",
       " 0.235  0.318  0.373  0.314  0.322  …  0.365  0.475  0.424  0.251  0.353\n",
       " 0.224  0.263  0.263  0.282  0.365     0.251  0.318  0.22   0.192  0.443\n",
       " 0.306  0.337  0.337  0.392  0.38      0.133  0.306  0.102  0.188  0.498\n",
       " 0.376  0.373  0.365  0.357  0.286     0.129  0.106  0.055  0.282  0.51\n",
       " 0.373  0.388  0.322  0.306  0.286     0.424  0.302  0.133  0.365  0.522\n",
       " 0.353  0.314  0.275  0.259  0.275  …  0.384  0.259  0.208  0.447  0.525\n",
       " 0.318  0.286  0.271  0.286  0.349     0.482  0.345  0.325  0.475  0.522\n",
       "\n",
       "[:, :, 3, 1] =\n",
       " 0.733  0.663  0.643  0.608  0.565  …  0.667  0.627  0.651  0.502  0.471\n",
       " 0.533  0.604  0.584  0.573  0.596     0.588  0.596  0.627  0.51   0.478\n",
       " 0.373  0.463  0.439  0.475  0.557     0.22   0.518  0.667  0.557  0.522\n",
       " 0.388  0.518  0.624  0.545  0.745     0.188  0.475  0.639  0.498  0.49\n",
       " 0.545  0.678  0.635  0.655  0.89      0.224  0.459  0.647  0.373  0.125\n",
       " 0.706  0.686  0.604  0.776  0.875  …  0.208  0.451  0.671  0.408  0.047\n",
       " 0.824  0.784  0.745  0.878  0.765     0.173  0.463  0.706  0.471  0.075\n",
       " 0.839  0.769  0.753  0.902  0.725     0.18   0.486  0.729  0.537  0.275\n",
       " 0.612  0.537  0.686  0.882  0.663     0.294  0.525  0.682  0.596  0.478\n",
       " 0.431  0.4    0.741  0.851  0.588     0.659  0.624  0.627  0.604  0.42\n",
       " 0.384  0.471  0.851  0.776  0.522  …  0.761  0.612  0.667  0.702  0.388\n",
       " 0.4    0.612  0.933  0.769  0.604     0.537  0.522  0.675  0.702  0.357\n",
       " 0.459  0.733  0.922  0.745  0.576     0.424  0.455  0.596  0.529  0.243\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.404  0.533  0.698  0.753  0.573  …  0.392  0.361  0.302  0.173  0.043\n",
       " 0.325  0.333  0.38   0.486  0.514     0.361  0.376  0.271  0.118  0.047\n",
       " 0.298  0.329  0.361  0.341  0.396     0.349  0.361  0.224  0.071  0.086\n",
       " 0.31   0.341  0.322  0.361  0.392     0.384  0.376  0.184  0.035  0.165\n",
       " 0.271  0.337  0.388  0.345  0.329     0.412  0.396  0.224  0.078  0.263\n",
       " 0.239  0.302  0.337  0.259  0.259  …  0.416  0.529  0.478  0.302  0.396\n",
       " 0.212  0.235  0.212  0.216  0.286     0.263  0.341  0.251  0.227  0.478\n",
       " 0.282  0.298  0.275  0.314  0.29      0.129  0.306  0.114  0.204  0.522\n",
       " 0.329  0.314  0.294  0.275  0.196     0.145  0.114  0.055  0.286  0.533\n",
       " 0.278  0.306  0.251  0.251  0.235     0.482  0.329  0.141  0.376  0.545\n",
       " 0.278  0.243  0.216  0.208  0.224  …  0.435  0.286  0.224  0.471  0.557\n",
       " 0.275  0.239  0.216  0.231  0.29      0.522  0.369  0.357  0.514  0.565\n",
       "\n",
       "[:, :, 1, 2] =\n",
       " 1.0    1.0    1.0    1.0    1.0    …  0.384  0.439  0.443  0.435  0.416\n",
       " 0.992  1.0    0.996  1.0    1.0       0.392  0.431  0.435  0.408  0.388\n",
       " 0.992  1.0    0.996  1.0    1.0       0.38   0.431  0.412  0.388  0.373\n",
       " 0.992  1.0    0.996  1.0    1.0       0.361  0.404  0.373  0.349  0.349\n",
       " 0.992  1.0    0.996  1.0    1.0       0.349  0.349  0.329  0.306  0.322\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.318  0.333  0.337  0.282  0.306\n",
       " 0.992  1.0    0.996  1.0    1.0       0.302  0.329  0.349  0.275  0.31\n",
       " 0.992  1.0    0.996  1.0    1.0       0.306  0.318  0.325  0.278  0.322\n",
       " 0.992  1.0    0.996  1.0    1.0       0.298  0.306  0.294  0.29   0.329\n",
       " 0.992  1.0    0.996  1.0    1.0       0.275  0.294  0.267  0.298  0.325\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.267  0.278  0.271  0.306  0.306\n",
       " 0.992  1.0    0.996  1.0    1.0       0.267  0.271  0.275  0.294  0.31\n",
       " 0.992  1.0    0.996  1.0    1.0       0.278  0.271  0.282  0.29   0.298\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.98   0.945  0.91   0.835  0.678  …  0.267  0.259  0.255  0.259  0.282\n",
       " 0.992  1.0    1.0    1.0    0.929     0.278  0.243  0.251  0.267  0.251\n",
       " 0.988  1.0    0.992  0.992  0.992     0.263  0.224  0.243  0.278  0.251\n",
       " 0.98   0.992  0.988  0.984  0.984     0.243  0.216  0.243  0.275  0.271\n",
       " 0.992  1.0    0.996  1.0    0.996     0.224  0.216  0.239  0.247  0.259\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.231  0.22   0.235  0.208  0.216\n",
       " 0.992  1.0    0.996  1.0    1.0       0.235  0.212  0.224  0.212  0.204\n",
       " 0.992  1.0    0.996  1.0    1.0       0.231  0.208  0.216  0.243  0.227\n",
       " 0.992  1.0    0.996  1.0    1.0       0.239  0.247  0.247  0.259  0.267\n",
       " 0.992  1.0    0.996  1.0    1.0       0.251  0.275  0.282  0.267  0.306\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.259  0.282  0.282  0.275  0.31\n",
       " 0.992  1.0    0.996  1.0    1.0       0.263  0.298  0.282  0.306  0.314\n",
       "\n",
       "[:, :, 2, 2] =\n",
       " 1.0    1.0    1.0    1.0    1.0    …  0.416  0.471  0.471  0.463  0.443\n",
       " 0.992  1.0    0.996  1.0    1.0       0.416  0.459  0.463  0.435  0.416\n",
       " 0.992  1.0    0.996  1.0    1.0       0.404  0.455  0.439  0.416  0.4\n",
       " 0.992  1.0    0.996  1.0    1.0       0.388  0.431  0.404  0.38   0.376\n",
       " 0.992  1.0    0.996  1.0    1.0       0.392  0.392  0.373  0.345  0.361\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.361  0.38   0.384  0.325  0.353\n",
       " 0.992  1.0    0.996  1.0    1.0       0.345  0.38   0.392  0.322  0.353\n",
       " 0.992  1.0    0.996  1.0    1.0       0.349  0.369  0.369  0.322  0.365\n",
       " 0.992  1.0    0.996  1.0    1.0       0.341  0.357  0.341  0.333  0.373\n",
       " 0.992  1.0    0.996  1.0    1.0       0.329  0.349  0.325  0.353  0.38\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.322  0.341  0.333  0.369  0.365\n",
       " 0.992  1.0    0.996  1.0    1.0       0.322  0.329  0.337  0.353  0.369\n",
       " 0.992  1.0    0.996  1.0    1.0       0.333  0.329  0.341  0.349  0.357\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.965  0.922  0.867  0.769  0.584  …  0.298  0.294  0.298  0.314  0.337\n",
       " 0.988  1.0    0.992  0.98   0.894     0.314  0.278  0.294  0.322  0.306\n",
       " 0.992  1.0    0.992  0.988  0.988     0.298  0.259  0.29   0.333  0.306\n",
       " 0.992  1.0    0.996  0.992  0.988     0.282  0.259  0.29   0.329  0.325\n",
       " 0.992  1.0    0.996  1.0    0.996     0.275  0.267  0.294  0.302  0.314\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.286  0.275  0.29   0.263  0.271\n",
       " 0.992  1.0    0.996  1.0    1.0       0.29   0.267  0.278  0.267  0.259\n",
       " 0.992  1.0    0.996  1.0    1.0       0.286  0.267  0.275  0.298  0.282\n",
       " 0.992  1.0    0.996  1.0    1.0       0.29   0.298  0.294  0.306  0.314\n",
       " 0.992  1.0    0.996  1.0    1.0       0.29   0.314  0.318  0.294  0.333\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.298  0.318  0.314  0.298  0.333\n",
       " 0.992  1.0    0.996  1.0    1.0       0.302  0.333  0.314  0.329  0.337\n",
       "\n",
       "[:, :, 3, 2] =\n",
       " 1.0    1.0    1.0    1.0    1.0    …  0.376  0.431  0.439  0.431  0.412\n",
       " 0.992  1.0    0.996  1.0    1.0       0.396  0.435  0.435  0.408  0.384\n",
       " 0.992  1.0    0.996  1.0    1.0       0.392  0.443  0.416  0.384  0.369\n",
       " 0.992  1.0    0.996  1.0    1.0       0.384  0.424  0.38   0.349  0.345\n",
       " 0.992  1.0    0.996  1.0    1.0       0.38   0.376  0.353  0.325  0.341\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.345  0.357  0.365  0.31   0.337\n",
       " 0.992  1.0    0.996  1.0    1.0       0.329  0.357  0.373  0.302  0.337\n",
       " 0.992  1.0    0.996  1.0    1.0       0.337  0.345  0.349  0.306  0.349\n",
       " 0.992  1.0    0.996  1.0    1.0       0.329  0.337  0.322  0.318  0.357\n",
       " 0.992  1.0    0.996  1.0    1.0       0.322  0.337  0.31   0.337  0.361\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.318  0.329  0.314  0.349  0.345\n",
       " 0.992  1.0    0.996  1.0    1.0       0.318  0.318  0.322  0.333  0.349\n",
       " 0.992  1.0    0.996  1.0    1.0       0.329  0.318  0.325  0.329  0.337\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.953  0.906  0.847  0.741  0.549  …  0.322  0.314  0.306  0.314  0.337\n",
       " 0.98   0.984  0.976  0.965  0.878     0.333  0.298  0.302  0.322  0.306\n",
       " 0.988  0.996  0.988  0.988  0.984     0.318  0.278  0.298  0.333  0.306\n",
       " 0.992  1.0    0.996  0.996  0.992     0.302  0.275  0.298  0.329  0.325\n",
       " 0.992  1.0    0.996  1.0    0.996     0.286  0.278  0.298  0.302  0.314\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.298  0.286  0.294  0.263  0.271\n",
       " 0.992  1.0    0.996  1.0    1.0       0.302  0.278  0.282  0.267  0.259\n",
       " 0.992  1.0    0.996  1.0    1.0       0.298  0.275  0.278  0.298  0.282\n",
       " 0.992  1.0    0.996  1.0    1.0       0.298  0.306  0.298  0.306  0.31\n",
       " 0.992  1.0    0.996  1.0    1.0       0.294  0.318  0.314  0.286  0.325\n",
       " 0.992  1.0    0.996  1.0    1.0    …  0.302  0.322  0.31   0.294  0.325\n",
       " 0.992  1.0    0.996  1.0    1.0       0.306  0.337  0.31   0.322  0.329\n",
       "\n",
       "[:, :, 1, 3] =\n",
       " 0.49   0.557  0.706  0.773  0.612  …  0.412  0.404  0.408  0.408  0.412\n",
       " 0.431  0.573  0.561  0.694  0.667     0.243  0.314  0.396  0.412  0.424\n",
       " 0.4    0.69   0.612  0.584  0.62      0.169  0.259  0.396  0.427  0.451\n",
       " 0.416  0.757  0.729  0.659  0.592     0.133  0.231  0.388  0.447  0.482\n",
       " 0.416  0.733  0.757  0.714  0.624     0.122  0.227  0.396  0.467  0.498\n",
       " 0.553  0.737  0.725  0.741  0.71   …  0.137  0.247  0.408  0.467  0.502\n",
       " 0.686  0.733  0.749  0.773  0.741     0.165  0.271  0.408  0.463  0.494\n",
       " 0.686  0.702  0.694  0.737  0.718     0.204  0.294  0.4    0.451  0.482\n",
       " 0.58   0.733  0.69   0.702  0.678     0.188  0.282  0.392  0.427  0.447\n",
       " 0.416  0.718  0.773  0.769  0.773     0.157  0.224  0.337  0.404  0.467\n",
       " 0.29   0.584  0.808  0.792  0.8    …  0.114  0.188  0.357  0.455  0.486\n",
       " 0.278  0.525  0.702  0.761  0.78      0.09   0.173  0.365  0.447  0.471\n",
       " 0.286  0.463  0.663  0.678  0.745     0.078  0.149  0.337  0.451  0.475\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.702  0.369  0.251  0.2    0.122  …  0.098  0.208  0.408  0.482  0.502\n",
       " 0.737  0.4    0.247  0.224  0.161     0.141  0.298  0.447  0.486  0.51\n",
       " 0.769  0.435  0.259  0.243  0.2       0.18   0.357  0.459  0.49   0.518\n",
       " 0.8    0.498  0.282  0.251  0.227     0.235  0.388  0.471  0.494  0.518\n",
       " 0.812  0.561  0.294  0.259  0.231     0.275  0.396  0.478  0.506  0.518\n",
       " 0.804  0.62   0.314  0.263  0.227  …  0.282  0.4    0.49   0.518  0.525\n",
       " 0.796  0.667  0.329  0.255  0.22      0.298  0.412  0.498  0.525  0.533\n",
       " 0.792  0.718  0.369  0.247  0.216     0.306  0.427  0.498  0.529  0.541\n",
       " 0.792  0.753  0.427  0.243  0.212     0.302  0.431  0.494  0.537  0.549\n",
       " 0.792  0.765  0.478  0.243  0.212     0.306  0.435  0.494  0.541  0.561\n",
       " 0.784  0.776  0.545  0.255  0.208  …  0.322  0.447  0.494  0.537  0.561\n",
       " 0.792  0.8    0.62   0.282  0.204     0.341  0.459  0.49   0.537  0.565\n",
       "\n",
       "[:, :, 2, 3] =\n",
       " 0.49   0.573  0.725  0.776  0.6    …  0.337  0.322  0.322  0.318  0.325\n",
       " 0.396  0.565  0.573  0.714  0.675     0.188  0.243  0.314  0.329  0.341\n",
       " 0.353  0.675  0.616  0.6    0.616     0.141  0.204  0.318  0.345  0.369\n",
       " 0.38   0.745  0.725  0.655  0.58      0.122  0.188  0.31   0.365  0.4\n",
       " 0.388  0.749  0.761  0.706  0.62      0.11   0.18   0.31   0.376  0.412\n",
       " 0.529  0.765  0.733  0.737  0.714  …  0.114  0.192  0.318  0.376  0.412\n",
       " 0.667  0.761  0.757  0.769  0.745     0.129  0.208  0.318  0.373  0.404\n",
       " 0.663  0.729  0.706  0.729  0.725     0.157  0.216  0.31   0.361  0.392\n",
       " 0.553  0.757  0.702  0.698  0.682     0.141  0.208  0.302  0.337  0.353\n",
       " 0.345  0.71   0.788  0.776  0.769     0.149  0.192  0.263  0.31   0.357\n",
       " 0.212  0.561  0.816  0.8    0.792  …  0.118  0.169  0.29   0.365  0.376\n",
       " 0.22   0.494  0.698  0.757  0.776     0.09   0.149  0.298  0.361  0.365\n",
       " 0.239  0.424  0.639  0.671  0.737     0.078  0.125  0.267  0.361  0.373\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.671  0.286  0.149  0.125  0.078  …  0.082  0.157  0.314  0.384  0.396\n",
       " 0.718  0.337  0.153  0.137  0.094     0.102  0.227  0.349  0.388  0.404\n",
       " 0.757  0.388  0.18   0.145  0.106     0.122  0.263  0.361  0.392  0.408\n",
       " 0.792  0.463  0.204  0.145  0.122     0.161  0.286  0.369  0.396  0.412\n",
       " 0.816  0.533  0.196  0.133  0.137     0.2    0.302  0.38   0.408  0.42\n",
       " 0.816  0.596  0.22   0.141  0.141  …  0.212  0.306  0.388  0.416  0.427\n",
       " 0.816  0.655  0.247  0.145  0.141     0.227  0.318  0.396  0.424  0.431\n",
       " 0.824  0.714  0.29   0.145  0.145     0.235  0.333  0.396  0.427  0.439\n",
       " 0.824  0.761  0.365  0.149  0.145     0.231  0.337  0.396  0.435  0.447\n",
       " 0.812  0.788  0.435  0.161  0.137     0.235  0.345  0.404  0.443  0.459\n",
       " 0.804  0.804  0.502  0.169  0.133  …  0.247  0.357  0.404  0.443  0.455\n",
       " 0.816  0.827  0.576  0.196  0.129     0.271  0.369  0.396  0.439  0.455\n",
       "\n",
       "[:, :, 3, 3] =\n",
       " 0.455  0.557  0.718  0.769  0.584  …  0.212  0.18   0.161  0.157  0.165\n",
       " 0.357  0.545  0.573  0.718  0.671     0.11   0.129  0.153  0.161  0.176\n",
       " 0.325  0.667  0.616  0.596  0.604     0.082  0.102  0.149  0.169  0.196\n",
       " 0.369  0.745  0.725  0.635  0.549     0.063  0.082  0.137  0.18   0.216\n",
       " 0.376  0.765  0.765  0.694  0.612     0.051  0.086  0.165  0.208  0.235\n",
       " 0.514  0.788  0.741  0.729  0.718  …  0.063  0.106  0.18   0.212  0.239\n",
       " 0.651  0.784  0.765  0.761  0.749     0.082  0.125  0.18   0.208  0.231\n",
       " 0.647  0.749  0.714  0.722  0.729     0.114  0.137  0.173  0.196  0.22\n",
       " 0.533  0.773  0.71   0.698  0.694     0.098  0.133  0.165  0.18   0.188\n",
       " 0.325  0.714  0.808  0.812  0.808     0.11   0.118  0.141  0.18   0.224\n",
       " 0.192  0.565  0.839  0.839  0.835  …  0.082  0.098  0.161  0.22   0.227\n",
       " 0.192  0.494  0.714  0.78   0.796     0.067  0.086  0.165  0.196  0.184\n",
       " 0.208  0.424  0.651  0.678  0.733     0.063  0.078  0.153  0.208  0.188\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.682  0.263  0.102  0.075  0.047  …  0.047  0.078  0.165  0.216  0.22\n",
       " 0.722  0.306  0.106  0.094  0.067     0.047  0.137  0.204  0.22   0.231\n",
       " 0.761  0.353  0.129  0.106  0.086     0.051  0.153  0.212  0.227  0.235\n",
       " 0.804  0.435  0.153  0.106  0.102     0.078  0.165  0.22   0.231  0.243\n",
       " 0.839  0.537  0.157  0.086  0.114     0.11   0.173  0.224  0.251  0.263\n",
       " 0.843  0.604  0.18   0.09   0.114  …  0.122  0.176  0.239  0.263  0.275\n",
       " 0.843  0.659  0.208  0.094  0.114     0.137  0.192  0.259  0.282  0.294\n",
       " 0.843  0.714  0.243  0.086  0.114     0.145  0.212  0.267  0.298  0.31\n",
       " 0.843  0.765  0.333  0.098  0.106     0.137  0.216  0.263  0.302  0.318\n",
       " 0.839  0.804  0.443  0.137  0.086     0.137  0.212  0.263  0.306  0.322\n",
       " 0.831  0.82   0.514  0.149  0.082  …  0.145  0.224  0.271  0.314  0.329\n",
       " 0.839  0.843  0.588  0.176  0.078     0.153  0.231  0.267  0.318  0.337\n",
       "\n",
       "...\n",
       "\n",
       "[:, :, 1, 48] =\n",
       " 0.631  0.682  0.722  0.718  0.698  …  0.38   0.365  0.318  0.282  0.294\n",
       " 0.647  0.725  0.796  0.753  0.702     0.365  0.341  0.294  0.286  0.306\n",
       " 0.682  0.776  0.796  0.741  0.71      0.357  0.349  0.318  0.314  0.329\n",
       " 0.694  0.741  0.769  0.749  0.722     0.357  0.349  0.337  0.325  0.329\n",
       " 0.694  0.753  0.796  0.753  0.702     0.341  0.333  0.341  0.337  0.314\n",
       " 0.686  0.745  0.773  0.733  0.675  …  0.357  0.337  0.357  0.357  0.322\n",
       " 0.651  0.694  0.722  0.698  0.651     0.373  0.353  0.357  0.353  0.329\n",
       " 0.651  0.69   0.714  0.686  0.647     0.38   0.369  0.337  0.337  0.329\n",
       " 0.667  0.706  0.718  0.682  0.643     0.388  0.373  0.337  0.345  0.337\n",
       " 0.667  0.714  0.714  0.682  0.647     0.384  0.357  0.349  0.353  0.333\n",
       " 0.671  0.718  0.714  0.686  0.647  …  0.376  0.337  0.345  0.349  0.333\n",
       " 0.682  0.722  0.714  0.686  0.651     0.369  0.337  0.353  0.353  0.341\n",
       " 0.686  0.722  0.71   0.686  0.659     0.361  0.333  0.341  0.337  0.361\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.839  0.855  0.788  0.757  0.753  …  0.431  0.443  0.404  0.384  0.404\n",
       " 0.824  0.824  0.776  0.769  0.808     0.416  0.42   0.384  0.38   0.4\n",
       " 0.82   0.8    0.765  0.788  0.827     0.408  0.412  0.365  0.373  0.392\n",
       " 0.82   0.792  0.776  0.792  0.824     0.388  0.396  0.345  0.333  0.38\n",
       " 0.8    0.796  0.784  0.78   0.824     0.365  0.365  0.341  0.325  0.357\n",
       " 0.784  0.796  0.784  0.769  0.792  …  0.341  0.349  0.353  0.345  0.349\n",
       " 0.776  0.776  0.753  0.769  0.78      0.322  0.345  0.345  0.349  0.361\n",
       " 0.769  0.765  0.745  0.776  0.78      0.302  0.341  0.329  0.349  0.365\n",
       " 0.737  0.761  0.761  0.749  0.729     0.278  0.341  0.322  0.361  0.369\n",
       " 0.722  0.765  0.769  0.722  0.667     0.255  0.337  0.314  0.329  0.333\n",
       " 0.729  0.749  0.749  0.686  0.651  …  0.286  0.329  0.298  0.325  0.318\n",
       " 0.745  0.733  0.698  0.635  0.627     0.333  0.322  0.314  0.322  0.314\n",
       "\n",
       "[:, :, 2, 48] =\n",
       " 0.682  0.706  0.733  0.733  0.725  …  0.369  0.357  0.337  0.31   0.306\n",
       " 0.686  0.737  0.792  0.753  0.718     0.353  0.333  0.31   0.31   0.314\n",
       " 0.71   0.78   0.784  0.733  0.718     0.345  0.341  0.333  0.337  0.337\n",
       " 0.71   0.737  0.753  0.741  0.737     0.361  0.353  0.353  0.345  0.337\n",
       " 0.706  0.749  0.784  0.757  0.725     0.353  0.345  0.353  0.345  0.322\n",
       " 0.71   0.753  0.773  0.749  0.71   …  0.369  0.349  0.369  0.369  0.325\n",
       " 0.686  0.714  0.733  0.722  0.698     0.384  0.365  0.369  0.365  0.333\n",
       " 0.686  0.71   0.729  0.722  0.702     0.38   0.373  0.353  0.353  0.337\n",
       " 0.69   0.722  0.733  0.718  0.706     0.376  0.369  0.357  0.369  0.345\n",
       " 0.69   0.729  0.729  0.718  0.71      0.38   0.361  0.365  0.369  0.337\n",
       " 0.69   0.733  0.733  0.722  0.71   …  0.376  0.353  0.357  0.353  0.333\n",
       " 0.706  0.737  0.733  0.722  0.714     0.38   0.369  0.369  0.357  0.345\n",
       " 0.71   0.737  0.729  0.725  0.718     0.384  0.376  0.373  0.353  0.365\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.816  0.824  0.761  0.737  0.745  …  0.404  0.412  0.392  0.376  0.38\n",
       " 0.792  0.792  0.745  0.737  0.773     0.404  0.396  0.373  0.373  0.365\n",
       " 0.792  0.769  0.733  0.757  0.796     0.404  0.4    0.369  0.38   0.373\n",
       " 0.788  0.761  0.741  0.761  0.792     0.396  0.392  0.365  0.357  0.373\n",
       " 0.776  0.773  0.761  0.757  0.8       0.388  0.388  0.369  0.345  0.357\n",
       " 0.769  0.78   0.769  0.753  0.776  …  0.384  0.396  0.384  0.361  0.361\n",
       " 0.761  0.761  0.737  0.753  0.769     0.361  0.392  0.376  0.365  0.373\n",
       " 0.753  0.749  0.729  0.761  0.765     0.337  0.384  0.365  0.369  0.373\n",
       " 0.725  0.749  0.749  0.741  0.722     0.31   0.384  0.361  0.388  0.384\n",
       " 0.71   0.761  0.765  0.725  0.678     0.29   0.38   0.357  0.376  0.376\n",
       " 0.718  0.749  0.749  0.694  0.663  …  0.325  0.373  0.341  0.369  0.365\n",
       " 0.733  0.733  0.698  0.643  0.643     0.376  0.369  0.361  0.361  0.357\n",
       "\n",
       "[:, :, 3, 48] =\n",
       " 0.722  0.729  0.741  0.745  0.749  …  0.122  0.118  0.098  0.075  0.078\n",
       " 0.718  0.749  0.792  0.757  0.733     0.106  0.098  0.078  0.078  0.09\n",
       " 0.733  0.784  0.773  0.725  0.722     0.102  0.106  0.098  0.106  0.118\n",
       " 0.741  0.749  0.749  0.741  0.749     0.102  0.102  0.098  0.098  0.106\n",
       " 0.753  0.773  0.792  0.765  0.741     0.09   0.082  0.09   0.09   0.082\n",
       " 0.765  0.784  0.788  0.765  0.733  …  0.106  0.086  0.106  0.114  0.09\n",
       " 0.749  0.753  0.757  0.749  0.733     0.122  0.102  0.106  0.106  0.098\n",
       " 0.745  0.749  0.757  0.753  0.741     0.118  0.114  0.094  0.098  0.094\n",
       " 0.741  0.757  0.761  0.753  0.749     0.118  0.118  0.106  0.118  0.102\n",
       " 0.745  0.765  0.757  0.749  0.753     0.129  0.11   0.114  0.118  0.094\n",
       " 0.745  0.769  0.761  0.753  0.753  …  0.141  0.102  0.102  0.102  0.094\n",
       " 0.761  0.773  0.761  0.757  0.757     0.149  0.11   0.106  0.102  0.098\n",
       " 0.765  0.773  0.757  0.761  0.769     0.137  0.086  0.086  0.094  0.118\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.816  0.816  0.745  0.729  0.737  …  0.118  0.157  0.149  0.122  0.129\n",
       " 0.784  0.784  0.737  0.729  0.765     0.11   0.153  0.129  0.11   0.118\n",
       " 0.784  0.761  0.725  0.749  0.788     0.122  0.153  0.118  0.11   0.122\n",
       " 0.78   0.753  0.733  0.753  0.784     0.125  0.149  0.102  0.071  0.114\n",
       " 0.769  0.765  0.753  0.745  0.788     0.125  0.11   0.098  0.086  0.11\n",
       " 0.757  0.769  0.757  0.741  0.769  …  0.129  0.075  0.106  0.137  0.122\n",
       " 0.749  0.749  0.725  0.741  0.757     0.125  0.086  0.09   0.129  0.129\n",
       " 0.741  0.737  0.718  0.749  0.757     0.129  0.09   0.071  0.114  0.129\n",
       " 0.714  0.737  0.741  0.733  0.714     0.114  0.11   0.067  0.118  0.133\n",
       " 0.702  0.757  0.765  0.729  0.686     0.086  0.125  0.075  0.086  0.102\n",
       " 0.706  0.745  0.749  0.698  0.675  …  0.094  0.106  0.071  0.094  0.094\n",
       " 0.722  0.725  0.698  0.647  0.655     0.114  0.086  0.09   0.102  0.09\n",
       "\n",
       "[:, :, 1, 49] =\n",
       " 0.439  0.522  0.647  0.522  0.18   …  0.522  0.455  0.545  0.525  0.392\n",
       " 0.525  0.655  0.694  0.349  0.106     0.502  0.514  0.522  0.58   0.506\n",
       " 0.537  0.635  0.565  0.212  0.188     0.506  0.486  0.486  0.71   0.639\n",
       " 0.6    0.651  0.42   0.2    0.49      0.373  0.392  0.42   0.69   0.757\n",
       " 0.675  0.608  0.314  0.286  0.725     0.451  0.463  0.475  0.706  0.863\n",
       " 0.678  0.49   0.224  0.439  0.831  …  0.549  0.549  0.529  0.733  0.933\n",
       " 0.647  0.365  0.357  0.624  0.902     0.58   0.576  0.557  0.737  0.965\n",
       " 0.584  0.443  0.494  0.588  0.922     0.604  0.6    0.58   0.733  0.976\n",
       " 0.537  0.455  0.447  0.49   0.808     0.592  0.592  0.576  0.729  0.973\n",
       " 0.49   0.404  0.447  0.427  0.478     0.557  0.565  0.553  0.718  0.969\n",
       " 0.455  0.388  0.435  0.412  0.408  …  0.592  0.588  0.573  0.753  0.976\n",
       " 0.431  0.376  0.408  0.396  0.4       0.576  0.573  0.569  0.78   0.988\n",
       " 0.424  0.361  0.4    0.388  0.4       0.525  0.525  0.557  0.816  1.0\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.753  0.706  0.498  0.576  0.596  …  0.922  0.714  0.51   0.494  0.482\n",
       " 0.788  0.729  0.443  0.412  0.471     0.89   0.627  0.498  0.494  0.482\n",
       " 0.855  0.725  0.533  0.376  0.478     0.835  0.565  0.494  0.498  0.482\n",
       " 0.922  0.733  0.627  0.353  0.467     0.769  0.514  0.494  0.482  0.475\n",
       " 0.976  0.784  0.69   0.396  0.455     0.682  0.498  0.51   0.502  0.49\n",
       " 1.0    0.847  0.733  0.49   0.42   …  0.612  0.482  0.478  0.447  0.427\n",
       " 1.0    0.925  0.769  0.588  0.392     0.518  0.455  0.404  0.345  0.325\n",
       " 1.0    0.98   0.8    0.671  0.357     0.459  0.392  0.349  0.306  0.286\n",
       " 1.0    1.0    0.859  0.722  0.561     0.318  0.212  0.133  0.098  0.082\n",
       " 1.0    0.996  0.922  0.749  0.808     0.106  0.098  0.094  0.09   0.086\n",
       " 1.0    1.0    0.98   0.812  0.82   …  0.11   0.098  0.098  0.106  0.11\n",
       " 0.996  0.984  0.992  0.867  0.78      0.098  0.165  0.247  0.271  0.251\n",
       "\n",
       "[:, :, 2, 49] =\n",
       " 0.294  0.322  0.529  0.561  0.2    …  0.518  0.459  0.573  0.475  0.141\n",
       " 0.357  0.455  0.627  0.384  0.114     0.475  0.49   0.525  0.592  0.31\n",
       " 0.376  0.553  0.576  0.22   0.192     0.467  0.451  0.482  0.741  0.557\n",
       " 0.475  0.647  0.431  0.212  0.49      0.337  0.353  0.404  0.706  0.745\n",
       " 0.612  0.627  0.329  0.306  0.729     0.4    0.416  0.439  0.702  0.867\n",
       " 0.675  0.506  0.231  0.459  0.835  …  0.49   0.494  0.482  0.725  0.937\n",
       " 0.663  0.376  0.38   0.647  0.902     0.514  0.518  0.502  0.722  0.969\n",
       " 0.588  0.471  0.549  0.62   0.918     0.537  0.533  0.518  0.714  0.984\n",
       " 0.541  0.502  0.51   0.525  0.816     0.537  0.529  0.514  0.714  0.984\n",
       " 0.506  0.455  0.498  0.475  0.514     0.498  0.498  0.49   0.698  0.984\n",
       " 0.475  0.435  0.475  0.459  0.443  …  0.525  0.518  0.514  0.729  0.988\n",
       " 0.451  0.42   0.451  0.439  0.435     0.51   0.506  0.51   0.757  0.992\n",
       " 0.447  0.408  0.443  0.431  0.439     0.455  0.455  0.498  0.796  1.0\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.753  0.702  0.498  0.596  0.616  …  0.918  0.486  0.078  0.078  0.071\n",
       " 0.792  0.729  0.451  0.443  0.51      0.859  0.314  0.059  0.082  0.071\n",
       " 0.859  0.729  0.529  0.396  0.522     0.757  0.188  0.063  0.086  0.075\n",
       " 0.929  0.745  0.624  0.365  0.51      0.616  0.098  0.078  0.078  0.075\n",
       " 0.98   0.792  0.69   0.404  0.49      0.459  0.067  0.106  0.09   0.086\n",
       " 1.0    0.855  0.733  0.494  0.451  …  0.302  0.067  0.075  0.071  0.075\n",
       " 1.0    0.929  0.769  0.588  0.42      0.2    0.153  0.165  0.173  0.196\n",
       " 0.996  0.98   0.804  0.675  0.369     0.333  0.318  0.298  0.271  0.247\n",
       " 0.996  0.996  0.863  0.722  0.565     0.278  0.176  0.118  0.086  0.071\n",
       " 0.992  0.992  0.922  0.753  0.812     0.094  0.082  0.086  0.086  0.078\n",
       " 1.0    0.996  0.98   0.82   0.824  …  0.102  0.094  0.09   0.086  0.09\n",
       " 0.988  0.976  0.988  0.871  0.784     0.094  0.122  0.11   0.09   0.09\n",
       "\n",
       "[:, :, 3, 49] =\n",
       " 0.227  0.216  0.451  0.545  0.192  …  0.498  0.455  0.58   0.482  0.149\n",
       " 0.271  0.341  0.569  0.388  0.098     0.4    0.455  0.529  0.592  0.314\n",
       " 0.29   0.506  0.561  0.216  0.18      0.404  0.42   0.467  0.733  0.557\n",
       " 0.404  0.627  0.424  0.212  0.49      0.286  0.318  0.376  0.69   0.733\n",
       " 0.569  0.612  0.329  0.31   0.741     0.349  0.369  0.4    0.678  0.847\n",
       " 0.655  0.498  0.239  0.471  0.851  …  0.435  0.435  0.439  0.702  0.91\n",
       " 0.647  0.38   0.392  0.675  0.922     0.459  0.459  0.455  0.702  0.949\n",
       " 0.584  0.486  0.565  0.643  0.937     0.482  0.475  0.467  0.694  0.965\n",
       " 0.549  0.522  0.529  0.541  0.831     0.471  0.467  0.459  0.686  0.969\n",
       " 0.518  0.471  0.51   0.49   0.529     0.431  0.435  0.431  0.675  0.973\n",
       " 0.49   0.451  0.49   0.475  0.463  …  0.455  0.451  0.451  0.706  0.984\n",
       " 0.467  0.435  0.463  0.455  0.455     0.443  0.439  0.447  0.733  0.992\n",
       " 0.455  0.42   0.459  0.447  0.451     0.396  0.384  0.435  0.78   1.0\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.741  0.678  0.475  0.604  0.624  …  0.898  0.482  0.086  0.09   0.075\n",
       " 0.78   0.702  0.435  0.459  0.525     0.847  0.314  0.067  0.09   0.075\n",
       " 0.855  0.706  0.514  0.404  0.533     0.741  0.192  0.071  0.09   0.078\n",
       " 0.929  0.725  0.596  0.361  0.525     0.608  0.098  0.082  0.082  0.078\n",
       " 0.984  0.78   0.663  0.384  0.506     0.459  0.075  0.106  0.09   0.082\n",
       " 1.0    0.855  0.714  0.471  0.463  …  0.294  0.063  0.078  0.075  0.075\n",
       " 1.0    0.937  0.757  0.569  0.42      0.212  0.149  0.157  0.153  0.169\n",
       " 1.0    0.984  0.796  0.655  0.361     0.322  0.298  0.263  0.231  0.208\n",
       " 1.0    1.0    0.859  0.702  0.549     0.259  0.169  0.102  0.082  0.075\n",
       " 0.996  0.992  0.925  0.745  0.804     0.094  0.09   0.086  0.082  0.078\n",
       " 1.0    1.0    0.976  0.816  0.816  …  0.098  0.098  0.094  0.09   0.086\n",
       " 0.992  0.98   0.988  0.867  0.776     0.098  0.118  0.106  0.098  0.09\n",
       "\n",
       "[:, :, 1, 50] =\n",
       " 0.898  0.886  0.894  0.902  0.914  …  0.051  0.016  0.004  0.035  0.294\n",
       " 0.847  0.831  0.843  0.859  0.878     0.016  0.008  0.004  0.004  0.157\n",
       " 0.902  0.882  0.882  0.882  0.882     0.031  0.008  0.004  0.0    0.09\n",
       " 0.914  0.902  0.918  0.925  0.929     0.051  0.008  0.0    0.0    0.094\n",
       " 0.878  0.867  0.882  0.886  0.902     0.063  0.008  0.008  0.016  0.11\n",
       " 0.894  0.875  0.875  0.878  0.886  …  0.114  0.024  0.02   0.031  0.137\n",
       " 0.929  0.906  0.914  0.922  0.906     0.204  0.086  0.035  0.031  0.165\n",
       " 0.808  0.8    0.851  0.878  0.835     0.212  0.153  0.145  0.114  0.227\n",
       " 0.6    0.631  0.737  0.749  0.694     0.31   0.192  0.2    0.212  0.286\n",
       " 0.604  0.553  0.675  0.671  0.667     0.404  0.349  0.271  0.255  0.282\n",
       " 0.753  0.506  0.561  0.549  0.62   …  0.408  0.463  0.471  0.435  0.392\n",
       " 0.902  0.624  0.424  0.42   0.478     0.463  0.416  0.49   0.561  0.584\n",
       " 0.922  0.773  0.38   0.282  0.345     0.51   0.427  0.471  0.588  0.639\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.91   0.91   0.788  0.439  0.388  …  0.612  0.62   0.51   0.31   0.176\n",
       " 0.878  0.875  0.773  0.384  0.392     0.565  0.6    0.522  0.294  0.196\n",
       " 0.875  0.871  0.729  0.31   0.345     0.651  0.671  0.647  0.49   0.384\n",
       " 0.855  0.835  0.58   0.357  0.435     0.69   0.702  0.698  0.714  0.663\n",
       " 0.898  0.8    0.486  0.518  0.561     0.702  0.706  0.706  0.761  0.745\n",
       " 0.871  0.675  0.537  0.678  0.675  …  0.682  0.71   0.733  0.741  0.737\n",
       " 0.812  0.592  0.639  0.749  0.792     0.682  0.698  0.749  0.784  0.722\n",
       " 0.827  0.694  0.78   0.867  0.886     0.69   0.682  0.733  0.812  0.718\n",
       " 0.863  0.851  0.875  0.882  0.851     0.694  0.69   0.745  0.773  0.714\n",
       " 0.875  0.89   0.894  0.902  0.882     0.694  0.698  0.761  0.773  0.729\n",
       " 0.867  0.878  0.894  0.875  0.843  …  0.698  0.698  0.725  0.769  0.745\n",
       " 0.886  0.894  0.906  0.898  0.89      0.702  0.682  0.718  0.69   0.729\n",
       "\n",
       "[:, :, 2, 50] =\n",
       " 0.91   0.898  0.906  0.914  0.925  …  0.051  0.016  0.004  0.035  0.298\n",
       " 0.867  0.851  0.859  0.878  0.898     0.016  0.008  0.004  0.004  0.161\n",
       " 0.918  0.894  0.898  0.898  0.898     0.027  0.004  0.004  0.0    0.082\n",
       " 0.925  0.914  0.929  0.937  0.941     0.047  0.004  0.0    0.0    0.086\n",
       " 0.898  0.886  0.898  0.902  0.914     0.063  0.008  0.008  0.02   0.102\n",
       " 0.91   0.89   0.894  0.894  0.902  …  0.11   0.031  0.02   0.035  0.129\n",
       " 0.941  0.922  0.925  0.933  0.922     0.192  0.086  0.035  0.031  0.176\n",
       " 0.824  0.816  0.863  0.89   0.847     0.192  0.153  0.141  0.11   0.243\n",
       " 0.608  0.627  0.737  0.757  0.69      0.29   0.184  0.188  0.22   0.318\n",
       " 0.604  0.549  0.678  0.682  0.667     0.384  0.318  0.251  0.251  0.298\n",
       " 0.769  0.502  0.553  0.553  0.616  …  0.384  0.424  0.412  0.373  0.349\n",
       " 0.918  0.62   0.408  0.408  0.467     0.439  0.38   0.42   0.459  0.498\n",
       " 0.937  0.788  0.376  0.263  0.329     0.467  0.4    0.4    0.467  0.518\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.918  0.918  0.816  0.42   0.365  …  0.557  0.565  0.435  0.259  0.165\n",
       " 0.894  0.886  0.796  0.361  0.369     0.529  0.584  0.49   0.271  0.2\n",
       " 0.886  0.886  0.741  0.282  0.322     0.635  0.663  0.639  0.478  0.38\n",
       " 0.863  0.855  0.592  0.329  0.416     0.682  0.706  0.706  0.714  0.659\n",
       " 0.902  0.812  0.486  0.494  0.545     0.702  0.722  0.718  0.765  0.753\n",
       " 0.882  0.675  0.514  0.655  0.663  …  0.694  0.722  0.745  0.741  0.745\n",
       " 0.824  0.576  0.616  0.737  0.784     0.69   0.71   0.753  0.776  0.729\n",
       " 0.839  0.686  0.776  0.867  0.875     0.698  0.694  0.733  0.8    0.729\n",
       " 0.871  0.863  0.882  0.878  0.839     0.702  0.702  0.753  0.78   0.725\n",
       " 0.886  0.898  0.902  0.894  0.875     0.706  0.706  0.765  0.776  0.737\n",
       " 0.875  0.882  0.894  0.878  0.855  …  0.706  0.706  0.737  0.773  0.753\n",
       " 0.894  0.894  0.906  0.898  0.886     0.718  0.698  0.733  0.698  0.737\n",
       "\n",
       "[:, :, 3, 50] =\n",
       " 0.925  0.914  0.925  0.941  0.953  …  0.039  0.008  0.004  0.039  0.294\n",
       " 0.851  0.831  0.839  0.859  0.894     0.016  0.008  0.004  0.004  0.157\n",
       " 0.929  0.906  0.91   0.898  0.894     0.02   0.004  0.004  0.0    0.082\n",
       " 0.949  0.937  0.961  0.969  0.973     0.031  0.004  0.004  0.0    0.067\n",
       " 0.882  0.871  0.886  0.902  0.929     0.047  0.004  0.008  0.02   0.071\n",
       " 0.914  0.89   0.886  0.886  0.91   …  0.086  0.02   0.012  0.031  0.098\n",
       " 0.961  0.929  0.953  0.969  0.941     0.149  0.078  0.031  0.024  0.141\n",
       " 0.765  0.757  0.831  0.871  0.82      0.161  0.118  0.122  0.098  0.216\n",
       " 0.514  0.576  0.686  0.69   0.62      0.247  0.157  0.169  0.208  0.314\n",
       " 0.541  0.502  0.627  0.616  0.592     0.318  0.263  0.22   0.239  0.29\n",
       " 0.698  0.416  0.471  0.467  0.541  …  0.325  0.329  0.314  0.29   0.251\n",
       " 0.906  0.565  0.31   0.325  0.396     0.365  0.282  0.263  0.259  0.275\n",
       " 0.945  0.773  0.286  0.2    0.278     0.357  0.29   0.243  0.224  0.259\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.843  0.839  0.718  0.31   0.263  …  0.353  0.361  0.263  0.173  0.133\n",
       " 0.816  0.808  0.678  0.263  0.278     0.322  0.408  0.357  0.192  0.157\n",
       " 0.808  0.82   0.643  0.196  0.239     0.455  0.529  0.514  0.38   0.306\n",
       " 0.773  0.761  0.498  0.239  0.302     0.541  0.576  0.58   0.608  0.553\n",
       " 0.816  0.718  0.373  0.376  0.424     0.588  0.596  0.584  0.639  0.627\n",
       " 0.78   0.561  0.404  0.553  0.537  …  0.588  0.608  0.6    0.576  0.62\n",
       " 0.718  0.455  0.522  0.62   0.639     0.6    0.616  0.651  0.667  0.616\n",
       " 0.741  0.588  0.686  0.722  0.737     0.627  0.6    0.573  0.694  0.62\n",
       " 0.784  0.776  0.796  0.765  0.698     0.639  0.616  0.604  0.663  0.616\n",
       " 0.8    0.816  0.82   0.804  0.722     0.643  0.608  0.667  0.686  0.627\n",
       " 0.808  0.839  0.863  0.82   0.745  …  0.647  0.6    0.624  0.694  0.659\n",
       " 0.804  0.816  0.835  0.8    0.776     0.667  0.62   0.651  0.635  0.655"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_batches[1] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A convolutional model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin with a convolutional model that has two convolutional layers, some batch normalization, maxpooling, and dropout for the dense layers that follow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6-element Array{Any,1}:\n",
       " AbstractFloat\n",
       " AbstractIrrational\n",
       " FixedPointNumbers.FixedPoint\n",
       " ForwardDiff.Dual\n",
       " Integer\n",
       " Rational"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subtypes(Real)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Any"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "supertype(typeof(352 => \"hello\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pair{Int64,String}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "typeof(352 => \"hello\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "search: \u001b[0m\u001b[1mC\u001b[22m\u001b[0m\u001b[1mo\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1mv\u001b[22m \u001b[0m\u001b[1mc\u001b[22m\u001b[0m\u001b[1mo\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1mv\u001b[22m \u001b[0m\u001b[1mc\u001b[22m\u001b[0m\u001b[1mo\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1mv\u001b[22m! \u001b[0m\u001b[1mc\u001b[22m\u001b[0m\u001b[1mo\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1mv\u001b[22mert \u001b[0m\u001b[1mC\u001b[22m\u001b[0m\u001b[1mo\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1mv\u001b[22mDims \u001b[0m\u001b[1mC\u001b[22m\u001b[0m\u001b[1mo\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1mv\u001b[22mTranspose \u001b[0m\u001b[1mc\u001b[22m\u001b[0m\u001b[1mo\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1mv\u001b[22m_bias_act\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/latex": [
       "\\begin{verbatim}\n",
       "Conv(filter, in => out, σ = identity; init = glorot_uniform,\n",
       "     stride = 1, pad = 0, dilation = 1)\n",
       "\n",
       "filter = (2,2)\n",
       "in = 1\n",
       "out = 16\n",
       "Conv((2, 2), 1=>16, relu)\n",
       "\\end{verbatim}\n",
       "Standard convolutional layer. \\texttt{filter} should be a tuple like \\texttt{(2, 2)}. \\texttt{in} and \\texttt{out} specify the number of input and output channels respectively.\n",
       "\n",
       "Data should be stored in WHCN order (width, height, \\# channels, batch size). In other words, a 100×100 RGB image would be a \\texttt{100×100×3×1} array, and a batch of 50 would be a \\texttt{100×100×3×50} array.\n",
       "\n",
       "Accepts keyword arguments \\texttt{weight} and \\texttt{bias} to set the corresponding fields. Setting \\texttt{bias} to \\texttt{Flux.Zeros()} will switch bias off for the layer.\n",
       "\n",
       "Takes the keyword arguments \\texttt{pad}, \\texttt{stride} and \\texttt{dilation}. For input dimension N,  \\texttt{pad} should be a single Integer indicating equal padding value for each spatial dimension, a tuple of length (N-2) to apply symmetric padding or a tuple of length 2*(N-2) indicating padding values for each spatial dimension at both the ends. \\texttt{stride} and \\texttt{dilation} should be a single Integer or a tuple with N-2 parameters. Use \\texttt{pad=SamePad()} to apply padding so that outputsize == inputsize / stride.\n",
       "\n",
       "\\section{Examples}\n",
       "Apply a \\texttt{Conv} layer to a 1-channel input using a 2×2 window filter size, giving us a 16-channel output. Output is activated with ReLU.\n",
       "\n",
       "\\begin{verbatim}\n",
       "filter = (2,2)\n",
       "in = 1\n",
       "out = 16\n",
       "Conv(filter, in => out, relu)\n",
       "\\end{verbatim}\n",
       "\\rule{\\textwidth}{1pt}\n",
       "\\begin{verbatim}\n",
       "Conv(weight::AbstractArray, bias::AbstractArray)\n",
       "Conv(weight::AbstractArray, bias::AbstractArray, activation)\n",
       "\\end{verbatim}\n",
       "Constructs the convolutional layer with user defined weight and bias arrays.\n",
       "\n",
       "Setting \\texttt{bias} to \\texttt{Flux.Zeros()} would switch \\texttt{bias} off for the layer.\n",
       "\n",
       "Takes the keyword arguments \\texttt{pad}, \\texttt{stride} and \\texttt{dilation}. For input dimension N,  \\texttt{pad} should be a single Integer indicating equal padding value for each spatial dimension, a tuple of length (N-2) to apply symmetric padding or a tuple of length 2*(N-2) indicating padding values for each spatial dimension at both the ends. \\texttt{stride} and \\texttt{dilation} should be a single Integer or a tuple with N-2 parameters.\n",
       "\n",
       "There is also a keyword-only constuctor available for all convoultional layers.\n",
       "\n",
       "\\begin{verbatim}\n",
       "weight = rand(Float32, 3, 3, 5)\n",
       "bias = zeros(Float32, 5)\n",
       "Conv(weight = weight,\n",
       "    bias = bias,\n",
       "    σ = sigmoid)\n",
       "\\end{verbatim}\n"
      ],
      "text/markdown": [
       "```\n",
       "Conv(filter, in => out, σ = identity; init = glorot_uniform,\n",
       "     stride = 1, pad = 0, dilation = 1)\n",
       "\n",
       "filter = (2,2)\n",
       "in = 1\n",
       "out = 16\n",
       "Conv((2, 2), 1=>16, relu)\n",
       "```\n",
       "\n",
       "Standard convolutional layer. `filter` should be a tuple like `(2, 2)`. `in` and `out` specify the number of input and output channels respectively.\n",
       "\n",
       "Data should be stored in WHCN order (width, height, # channels, batch size). In other words, a 100×100 RGB image would be a `100×100×3×1` array, and a batch of 50 would be a `100×100×3×50` array.\n",
       "\n",
       "Accepts keyword arguments `weight` and `bias` to set the corresponding fields. Setting `bias` to `Flux.Zeros()` will switch bias off for the layer.\n",
       "\n",
       "Takes the keyword arguments `pad`, `stride` and `dilation`. For input dimension N,  `pad` should be a single Integer indicating equal padding value for each spatial dimension, a tuple of length (N-2) to apply symmetric padding or a tuple of length 2*(N-2) indicating padding values for each spatial dimension at both the ends. `stride` and `dilation` should be a single Integer or a tuple with N-2 parameters. Use `pad=SamePad()` to apply padding so that outputsize == inputsize / stride.\n",
       "\n",
       "# Examples\n",
       "\n",
       "Apply a `Conv` layer to a 1-channel input using a 2×2 window filter size, giving us a 16-channel output. Output is activated with ReLU.\n",
       "\n",
       "```julia\n",
       "filter = (2,2)\n",
       "in = 1\n",
       "out = 16\n",
       "Conv(filter, in => out, relu)\n",
       "```\n",
       "\n",
       "---\n",
       "\n",
       "```\n",
       "Conv(weight::AbstractArray, bias::AbstractArray)\n",
       "Conv(weight::AbstractArray, bias::AbstractArray, activation)\n",
       "```\n",
       "\n",
       "Constructs the convolutional layer with user defined weight and bias arrays.\n",
       "\n",
       "Setting `bias` to `Flux.Zeros()` would switch `bias` off for the layer.\n",
       "\n",
       "Takes the keyword arguments `pad`, `stride` and `dilation`. For input dimension N,  `pad` should be a single Integer indicating equal padding value for each spatial dimension, a tuple of length (N-2) to apply symmetric padding or a tuple of length 2*(N-2) indicating padding values for each spatial dimension at both the ends. `stride` and `dilation` should be a single Integer or a tuple with N-2 parameters.\n",
       "\n",
       "There is also a keyword-only constuctor available for all convoultional layers.\n",
       "\n",
       "```julia\n",
       "weight = rand(Float32, 3, 3, 5)\n",
       "bias = zeros(Float32, 5)\n",
       "Conv(weight = weight,\n",
       "    bias = bias,\n",
       "    σ = sigmoid)\n",
       "```\n"
      ],
      "text/plain": [
       "\u001b[36m  Conv(filter, in => out, σ = identity; init = glorot_uniform,\u001b[39m\n",
       "\u001b[36m       stride = 1, pad = 0, dilation = 1)\u001b[39m\n",
       "\u001b[36m  \u001b[39m\n",
       "\u001b[36m  filter = (2,2)\u001b[39m\n",
       "\u001b[36m  in = 1\u001b[39m\n",
       "\u001b[36m  out = 16\u001b[39m\n",
       "\u001b[36m  Conv((2, 2), 1=>16, relu)\u001b[39m\n",
       "\n",
       "  Standard convolutional layer. \u001b[36mfilter\u001b[39m should be a tuple like \u001b[36m(2, 2)\u001b[39m. \u001b[36min\u001b[39m and\n",
       "  \u001b[36mout\u001b[39m specify the number of input and output channels respectively.\n",
       "\n",
       "  Data should be stored in WHCN order (width, height, # channels, batch size).\n",
       "  In other words, a 100×100 RGB image would be a \u001b[36m100×100×3×1\u001b[39m array, and a\n",
       "  batch of 50 would be a \u001b[36m100×100×3×50\u001b[39m array.\n",
       "\n",
       "  Accepts keyword arguments \u001b[36mweight\u001b[39m and \u001b[36mbias\u001b[39m to set the corresponding fields.\n",
       "  Setting \u001b[36mbias\u001b[39m to \u001b[36mFlux.Zeros()\u001b[39m will switch bias off for the layer.\n",
       "\n",
       "  Takes the keyword arguments \u001b[36mpad\u001b[39m, \u001b[36mstride\u001b[39m and \u001b[36mdilation\u001b[39m. For input dimension N,\n",
       "  \u001b[36mpad\u001b[39m should be a single Integer indicating equal padding value for each\n",
       "  spatial dimension, a tuple of length (N-2) to apply symmetric padding or a\n",
       "  tuple of length 2*(N-2) indicating padding values for each spatial dimension\n",
       "  at both the ends. \u001b[36mstride\u001b[39m and \u001b[36mdilation\u001b[39m should be a single Integer or a tuple\n",
       "  with N-2 parameters. Use \u001b[36mpad=SamePad()\u001b[39m to apply padding so that outputsize\n",
       "  == inputsize / stride.\n",
       "\n",
       "\u001b[1m  Examples\u001b[22m\n",
       "\u001b[1m  ≡≡≡≡≡≡≡≡≡≡\u001b[22m\n",
       "\n",
       "  Apply a \u001b[36mConv\u001b[39m layer to a 1-channel input using a 2×2 window filter size,\n",
       "  giving us a 16-channel output. Output is activated with ReLU.\n",
       "\n",
       "\u001b[36m  filter = (2,2)\u001b[39m\n",
       "\u001b[36m  in = 1\u001b[39m\n",
       "\u001b[36m  out = 16\u001b[39m\n",
       "\u001b[36m  Conv(filter, in => out, relu)\u001b[39m\n",
       "\n",
       "  ────────────────────────────────────────────────────────────────────────────\n",
       "\n",
       "\u001b[36m  Conv(weight::AbstractArray, bias::AbstractArray)\u001b[39m\n",
       "\u001b[36m  Conv(weight::AbstractArray, bias::AbstractArray, activation)\u001b[39m\n",
       "\n",
       "  Constructs the convolutional layer with user defined weight and bias arrays.\n",
       "\n",
       "  Setting \u001b[36mbias\u001b[39m to \u001b[36mFlux.Zeros()\u001b[39m would switch \u001b[36mbias\u001b[39m off for the layer.\n",
       "\n",
       "  Takes the keyword arguments \u001b[36mpad\u001b[39m, \u001b[36mstride\u001b[39m and \u001b[36mdilation\u001b[39m. For input dimension N,\n",
       "  \u001b[36mpad\u001b[39m should be a single Integer indicating equal padding value for each\n",
       "  spatial dimension, a tuple of length (N-2) to apply symmetric padding or a\n",
       "  tuple of length 2*(N-2) indicating padding values for each spatial dimension\n",
       "  at both the ends. \u001b[36mstride\u001b[39m and \u001b[36mdilation\u001b[39m should be a single Integer or a tuple\n",
       "  with N-2 parameters.\n",
       "\n",
       "  There is also a keyword-only constuctor available for all convoultional\n",
       "  layers.\n",
       "\n",
       "\u001b[36m  weight = rand(Float32, 3, 3, 5)\u001b[39m\n",
       "\u001b[36m  bias = zeros(Float32, 5)\u001b[39m\n",
       "\u001b[36m  Conv(weight = weight,\u001b[39m\n",
       "\u001b[36m      bias = bias,\u001b[39m\n",
       "\u001b[36m      σ = sigmoid)\u001b[39m"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "? Conv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{32 +2 \\times 1 - 3}{1} + 1 = 16\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Chain(Conv((3, 3), 3=>8, relu), BatchNorm(8), #27, Conv((3, 3), 8=>4, relu), BatchNorm(4), #28, flatten, Dense(196, 80, relu), Dropout(0.5), Dense(80, 40, relu), Dropout(0.5), Dense(40, 3), softmax)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Flux\n",
    "\n",
    "function buildModel1()\n",
    "    Chain(  #Assuming 32x32x3 input layer (like CIFAR10)\n",
    "      Conv((3, 3), 3 => 8, relu, pad=(1, 1), stride=(1, 1)),   #32x32x8 convolutional layer\n",
    "      BatchNorm(8),\n",
    "      x -> maxpool(x, (2, 2)),  #16x16x8\n",
    "      Conv((3, 3), 8 => 4, relu, pad=(0, 0), stride=(1, 1)), #14x14x4  convolutional layer\n",
    "      BatchNorm(4),\n",
    "      x -> maxpool(x, (2, 2)), #7x7x4 ,\n",
    "      flatten,  #196 neurons\n",
    "      Dense(196, 80, relu),\n",
    "      Dropout(0.5),\n",
    "      Dense(80, 40, relu), #40 neurons\n",
    "      Dropout(0.5),\n",
    "      Dense(40, 3), #3 output neruons\n",
    "      softmax)\n",
    "end\n",
    "\n",
    "model1 = buildModel1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To \"debug\" the model, it is good to consider one sample image from the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32×32×3×1 Array{N0f8,4} with eltype FixedPointNumbers.Normed{UInt8,8}:\n",
       "[:, :, 1, 1] =\n",
       " 0.604  0.549  0.549  0.533  0.506  …  0.682  0.643  0.686  0.647  0.639\n",
       " 0.494  0.569  0.545  0.537  0.553     0.553  0.553  0.612  0.612  0.62\n",
       " 0.412  0.49   0.451  0.478  0.533     0.173  0.451  0.604  0.624  0.639\n",
       " 0.4    0.486  0.576  0.518  0.729     0.169  0.443  0.576  0.514  0.569\n",
       " 0.49   0.588  0.541  0.592  0.843     0.224  0.455  0.608  0.369  0.169\n",
       " 0.608  0.596  0.518  0.71   0.792  …  0.204  0.447  0.631  0.4    0.075\n",
       " 0.675  0.682  0.667  0.796  0.643     0.173  0.443  0.627  0.424  0.078\n",
       " 0.706  0.698  0.698  0.816  0.588     0.188  0.455  0.655  0.502  0.29\n",
       " 0.557  0.525  0.671  0.816  0.541     0.31   0.486  0.647  0.604  0.525\n",
       " 0.435  0.431  0.753  0.796  0.467     0.678  0.584  0.596  0.612  0.467\n",
       " 0.416  0.522  0.859  0.702  0.369  …  0.851  0.627  0.639  0.714  0.431\n",
       " 0.427  0.639  0.918  0.663  0.424     0.651  0.557  0.643  0.702  0.388\n",
       " 0.482  0.753  0.898  0.643  0.424     0.38   0.384  0.522  0.49   0.239\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.455  0.557  0.694  0.718  0.522  …  0.31   0.271  0.243  0.137  0.024\n",
       " 0.4    0.376  0.396  0.475  0.49      0.286  0.278  0.2    0.082  0.039\n",
       " 0.373  0.388  0.396  0.357  0.4       0.282  0.271  0.173  0.055  0.098\n",
       " 0.353  0.373  0.345  0.369  0.384     0.329  0.294  0.153  0.043  0.2\n",
       " 0.282  0.349  0.404  0.357  0.345     0.341  0.31   0.169  0.055  0.267\n",
       " 0.235  0.314  0.369  0.302  0.314  …  0.369  0.463  0.4    0.231  0.353\n",
       " 0.22   0.255  0.255  0.271  0.353     0.282  0.337  0.216  0.192  0.455\n",
       " 0.302  0.329  0.325  0.38   0.369     0.169  0.333  0.125  0.212  0.525\n",
       " 0.369  0.361  0.353  0.345  0.271     0.133  0.122  0.09   0.318  0.549\n",
       " 0.357  0.376  0.31   0.298  0.278     0.404  0.302  0.165  0.404  0.561\n",
       " 0.341  0.302  0.267  0.251  0.267  …  0.373  0.267  0.239  0.482  0.561\n",
       " 0.31   0.278  0.263  0.278  0.341     0.486  0.369  0.365  0.514  0.561\n",
       "\n",
       "[:, :, 2, 1] =\n",
       " 0.694  0.627  0.608  0.576  0.537  …  0.643  0.608  0.655  0.604  0.58\n",
       " 0.537  0.6    0.573  0.557  0.573     0.537  0.545  0.604  0.596  0.58\n",
       " 0.408  0.49   0.451  0.475  0.545     0.18   0.467  0.627  0.631  0.612\n",
       " 0.396  0.506  0.6    0.522  0.729     0.161  0.443  0.6    0.51   0.529\n",
       " 0.514  0.631  0.588  0.616  0.863     0.2    0.424  0.6    0.345  0.125\n",
       " 0.651  0.643  0.569  0.757  0.847  …  0.18   0.412  0.604  0.361  0.035\n",
       " 0.745  0.737  0.722  0.871  0.737     0.145  0.424  0.639  0.42   0.055\n",
       " 0.78   0.741  0.741  0.89   0.698     0.145  0.439  0.678  0.506  0.267\n",
       " 0.612  0.545  0.69   0.875  0.635     0.271  0.482  0.647  0.584  0.49\n",
       " 0.471  0.435  0.765  0.859  0.573     0.675  0.612  0.596  0.592  0.431\n",
       " 0.42   0.498  0.855  0.761  0.478  …  0.792  0.612  0.635  0.694  0.396\n",
       " 0.408  0.612  0.914  0.722  0.537     0.537  0.502  0.639  0.686  0.365\n",
       " 0.475  0.753  0.929  0.729  0.545     0.361  0.392  0.541  0.506  0.247\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.459  0.561  0.694  0.718  0.525  …  0.314  0.278  0.251  0.145  0.024\n",
       " 0.396  0.38   0.4    0.482  0.502     0.29   0.29   0.208  0.086  0.035\n",
       " 0.373  0.396  0.404  0.369  0.42      0.275  0.278  0.169  0.047  0.086\n",
       " 0.349  0.376  0.349  0.38   0.408     0.306  0.298  0.141  0.024  0.176\n",
       " 0.275  0.349  0.404  0.369  0.361     0.333  0.322  0.173  0.051  0.251\n",
       " 0.235  0.318  0.373  0.314  0.322  …  0.365  0.475  0.424  0.251  0.353\n",
       " 0.224  0.263  0.263  0.282  0.365     0.251  0.318  0.22   0.192  0.443\n",
       " 0.306  0.337  0.337  0.392  0.38      0.133  0.306  0.102  0.188  0.498\n",
       " 0.376  0.373  0.365  0.357  0.286     0.129  0.106  0.055  0.282  0.51\n",
       " 0.373  0.388  0.322  0.306  0.286     0.424  0.302  0.133  0.365  0.522\n",
       " 0.353  0.314  0.275  0.259  0.275  …  0.384  0.259  0.208  0.447  0.525\n",
       " 0.318  0.286  0.271  0.286  0.349     0.482  0.345  0.325  0.475  0.522\n",
       "\n",
       "[:, :, 3, 1] =\n",
       " 0.733  0.663  0.643  0.608  0.565  …  0.667  0.627  0.651  0.502  0.471\n",
       " 0.533  0.604  0.584  0.573  0.596     0.588  0.596  0.627  0.51   0.478\n",
       " 0.373  0.463  0.439  0.475  0.557     0.22   0.518  0.667  0.557  0.522\n",
       " 0.388  0.518  0.624  0.545  0.745     0.188  0.475  0.639  0.498  0.49\n",
       " 0.545  0.678  0.635  0.655  0.89      0.224  0.459  0.647  0.373  0.125\n",
       " 0.706  0.686  0.604  0.776  0.875  …  0.208  0.451  0.671  0.408  0.047\n",
       " 0.824  0.784  0.745  0.878  0.765     0.173  0.463  0.706  0.471  0.075\n",
       " 0.839  0.769  0.753  0.902  0.725     0.18   0.486  0.729  0.537  0.275\n",
       " 0.612  0.537  0.686  0.882  0.663     0.294  0.525  0.682  0.596  0.478\n",
       " 0.431  0.4    0.741  0.851  0.588     0.659  0.624  0.627  0.604  0.42\n",
       " 0.384  0.471  0.851  0.776  0.522  …  0.761  0.612  0.667  0.702  0.388\n",
       " 0.4    0.612  0.933  0.769  0.604     0.537  0.522  0.675  0.702  0.357\n",
       " 0.459  0.733  0.922  0.745  0.576     0.424  0.455  0.596  0.529  0.243\n",
       " ⋮                                  ⋱                       ⋮      \n",
       " 0.404  0.533  0.698  0.753  0.573  …  0.392  0.361  0.302  0.173  0.043\n",
       " 0.325  0.333  0.38   0.486  0.514     0.361  0.376  0.271  0.118  0.047\n",
       " 0.298  0.329  0.361  0.341  0.396     0.349  0.361  0.224  0.071  0.086\n",
       " 0.31   0.341  0.322  0.361  0.392     0.384  0.376  0.184  0.035  0.165\n",
       " 0.271  0.337  0.388  0.345  0.329     0.412  0.396  0.224  0.078  0.263\n",
       " 0.239  0.302  0.337  0.259  0.259  …  0.416  0.529  0.478  0.302  0.396\n",
       " 0.212  0.235  0.212  0.216  0.286     0.263  0.341  0.251  0.227  0.478\n",
       " 0.282  0.298  0.275  0.314  0.29      0.129  0.306  0.114  0.204  0.522\n",
       " 0.329  0.314  0.294  0.275  0.196     0.145  0.114  0.055  0.286  0.533\n",
       " 0.278  0.306  0.251  0.251  0.235     0.482  0.329  0.141  0.376  0.545\n",
       " 0.278  0.243  0.216  0.208  0.224  …  0.435  0.286  0.224  0.471  0.557\n",
       " 0.275  0.239  0.216  0.231  0.29      0.522  0.369  0.357  0.514  0.565"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampleImage = x_train_batches[1][:,:,:,1:1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: Slow fallback implementation invoked for conv!  You probably don't want this; check your datatypes.\n",
      "│   yT = Float32\n",
      "│   T1 = FixedPointNumbers.Normed{UInt8,8}\n",
      "│   T2 = Float32\n",
      "└ @ NNlib /Users/uqjnazar/.julia/packages/NNlib/1WXEb/src/conv.jl:206\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3×1 Array{Float32,2}:\n",
       " 0.33271927\n",
       " 0.35835612\n",
       " 0.30892456"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#The output of the (untrained) modle on the sample image\n",
    "model1(sampleImage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length(model1) #The number of `layers`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16×16×8×1 Array{Float32,4}:\n",
       "[:, :, 1, 1] =\n",
       " 0.0          0.0        0.0         …  0.0        0.0        0.156001\n",
       " 0.0          0.0316348  0.052588       0.19134    0.0        0.144869\n",
       " 0.001296     0.0        0.17075        0.129435   0.0        0.215919\n",
       " 0.0          0.0        0.192647       0.177555   0.0        0.247374\n",
       " 0.0          0.0        0.146402       0.185332   0.0        0.16906\n",
       " 0.0          0.118273   0.199646    …  0.0        0.0227304  0.241958\n",
       " 0.0          0.366391   0.278903       0.0589004  0.0        0.21111\n",
       " 0.0          0.265847   0.194411       0.0329308  0.150364   0.163986\n",
       " 0.0          0.0        0.0526168      0.0        0.0722537  0.101766\n",
       " 0.0          0.0260685  0.0            0.0472933  0.0        0.109576\n",
       " 0.0          0.0105605  0.0194761   …  0.0185836  0.0569571  0.0853811\n",
       " 0.0          0.0        0.0            0.0371222  0.1118     0.0579876\n",
       " 0.0          0.0122142  0.0105474      0.0295492  0.174628   0.0463134\n",
       " 0.0          0.0        0.0            0.0        0.0764904  0.0223023\n",
       " 0.0          0.0        0.00795958     0.069635   0.0600813  0.042051\n",
       " 0.000891586  0.0163533  0.0         …  0.0        0.0930915  0.0891313\n",
       "\n",
       "[:, :, 2, 1] =\n",
       " 0.211378   0.0        0.0        …  0.00352297  0.027091   0.0\n",
       " 0.123039   0.0        0.0           0.0         0.0440005  0.0\n",
       " 0.0686048  0.0        0.0           0.0         0.0542466  0.0\n",
       " 0.234397   0.0        0.0           0.0         0.0367101  0.0\n",
       " 0.211714   0.0        0.0           0.0         0.0        0.0\n",
       " 0.0998725  0.0        0.0        …  0.0         0.0        0.0\n",
       " 0.135756   0.0        0.0667714     0.0196612   0.0977457  0.0\n",
       " 0.212242   0.0        0.0           0.0589471   0.0        0.0\n",
       " 0.259623   0.0        0.0           0.197879    0.16303    0.0\n",
       " 0.326148   0.0        0.0338633     0.0         0.0        0.0\n",
       " 0.23577    0.0138428  0.0174966  …  0.0         0.0        0.0\n",
       " 0.0956262  0.0        0.0           0.0         0.0        0.0\n",
       " 0.0997842  0.0        0.0           0.00886515  0.019869   0.0455283\n",
       " 0.0456238  0.0        0.0           0.0736498   0.0        0.0549446\n",
       " 0.105524   0.0        0.0           0.0179655   0.0        0.0100862\n",
       " 0.176237   0.184758   0.212655   …  0.380003    0.317877   0.367175\n",
       "\n",
       "[:, :, 3, 1] =\n",
       " 0.139865   0.0  0.0  0.0        …  0.0         0.0        0.0\n",
       " 0.225567   0.0  0.0  0.0           0.0         0.0703274  0.0\n",
       " 0.224653   0.0  0.0  0.0           0.0         0.0922245  0.0\n",
       " 0.185038   0.0  0.0  0.0           0.0         0.112722   0.0\n",
       " 0.125708   0.0  0.0  0.116034      0.0         0.0        0.0\n",
       " 0.251351   0.0  0.0  0.1246     …  0.0         0.0        0.0\n",
       " 0.367209   0.0  0.0  0.0472652     0.00135726  0.0        0.0\n",
       " 0.353666   0.0  0.0  0.0           0.0273802   0.0        0.0\n",
       " 0.262118   0.0  0.0  0.0           0.121527    0.0        0.0\n",
       " 0.275121   0.0  0.0  0.0           0.0         0.0        0.0\n",
       " 0.113297   0.0  0.0  0.0        …  0.0         0.0        0.0\n",
       " 0.105067   0.0  0.0  0.0           0.0         0.0        0.00375631\n",
       " 0.0745242  0.0  0.0  0.0           0.0         0.0        0.0564757\n",
       " 0.126298   0.0  0.0  0.0           0.0         0.0        0.148674\n",
       " 0.105765   0.0  0.0  0.0           0.084921    0.0661873  0.123603\n",
       " 0.0803138  0.0  0.0  0.0        …  0.0119629   0.0123489  0.0\n",
       "\n",
       "[:, :, 4, 1] =\n",
       " 0.202534  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0912649  0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0  …  0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0  …  0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0        0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0     0.0  0.0        0.0133206  0.0  0.0\n",
       " 0.0       0.0  0.0  0.0  0.0  0.0  …  0.0  0.0        0.0        0.0  0.0\n",
       "\n",
       "[:, :, 5, 1] =\n",
       " 0.118131    0.115859   0.0758447  …  0.18057      0.20197    0.240856\n",
       " 0.0904474   0.128774   0.0449249     0.229837     0.0923717  0.308818\n",
       " 0.0         0.0764449  0.271946      0.235984     0.0430414  0.353535\n",
       " 0.117578    0.0837194  0.272966      0.261533     0.0194808  0.265282\n",
       " 0.216011    0.085801   0.263174      0.287535     0.0550919  0.255565\n",
       " 0.0         0.208881   0.275952   …  0.211469     0.253988   0.370941\n",
       " 0.0         0.309069   0.270498      0.212113     0.131811   0.404038\n",
       " 0.0256245   0.332344   0.0465442     0.093619     0.117567   0.206005\n",
       " 0.133274    0.180203   0.218451      0.0397932    0.138337   0.0956695\n",
       " 0.0133247   0.213581   0.15587       0.0          0.188416   0.180373\n",
       " 0.089228    0.285134   0.20703    …  0.0699709    0.0845926  0.141033\n",
       " 0.0748954   0.0879569  0.0919954     0.0913016    0.120852   0.123201\n",
       " 0.0681801   0.124284   0.0896998     0.139392     0.117498   0.0143615\n",
       " 0.00620851  0.0914594  0.0764519     0.0819917    0.197149   0.151368\n",
       " 0.0652272   0.115888   0.121955      0.0682602    0.118961   0.0385595\n",
       " 0.133752    0.119092   0.0667756  …  0.000384373  0.16817    0.232341\n",
       "\n",
       "[:, :, 6, 1] =\n",
       " 0.449094  0.438641  0.426727  0.446886  …  0.521616  0.436801  0.530502\n",
       " 0.352909  0.381985  0.507432  0.428292     0.487602  0.441815  0.527641\n",
       " 0.431747  0.469577  0.609594  0.426339     0.548701  0.403866  0.45249\n",
       " 0.554922  0.593248  0.629282  0.393318     0.526349  0.401878  0.489725\n",
       " 0.433908  0.628056  0.621096  0.342874     0.499009  0.437103  0.550552\n",
       " 0.303907  0.680118  0.526477  0.313266  …  0.544466  0.500328  0.551211\n",
       " 0.439332  0.741259  0.49208   0.34716      0.370099  0.369228  0.460381\n",
       " 0.625972  0.770483  0.654236  0.486245     0.229262  0.233656  0.25049\n",
       " 0.625214  0.701368  0.703947  0.655879     0.238606  0.368228  0.288604\n",
       " 0.540377  0.681329  0.667274  0.708787     0.112909  0.273034  0.272811\n",
       " 0.398439  0.561434  0.500187  0.615838  …  0.30699   0.24623   0.189234\n",
       " 0.270867  0.275054  0.369809  0.375901     0.32898   0.256002  0.115547\n",
       " 0.246124  0.270879  0.262569  0.222286     0.288949  0.374768  0.198345\n",
       " 0.210724  0.234055  0.246689  0.241282     0.159933  0.281188  0.30253\n",
       " 0.261067  0.250447  0.229041  0.269852     0.218983  0.213784  0.39063\n",
       " 0.253793  0.240318  0.25964   0.257044  …  0.329033  0.346204  0.434382\n",
       "\n",
       "[:, :, 7, 1] =\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0159291\n",
       "\n",
       "[:, :, 8, 1] =\n",
       " 0.00512367  0.0         0.0        …  0.0        0.0123692   0.056147\n",
       " 0.0         0.00651896  0.013223      0.0509412  0.0240172   0.0382585\n",
       " 0.0         0.0         0.0           0.0194082  0.0         0.0312325\n",
       " 0.0         0.0         0.0392657     0.0470804  0.0         0.124582\n",
       " 0.0424623   0.0196358   0.0141826     0.137368   0.0         0.0381332\n",
       " 0.0         0.0         0.0428686  …  0.0        0.0         0.0696643\n",
       " 0.0         0.0813231   0.111288      0.0718753  0.0         0.110276\n",
       " 0.0         0.175933    0.106781      0.0119542  0.0723206   0.086491\n",
       " 0.0         0.0         0.0385427     0.0490721  0.00703945  0.0259346\n",
       " 0.0         0.0         0.0           0.0        0.0265809   0.0507335\n",
       " 0.0         0.0278393   0.047163   …  0.0        0.0         0.0541011\n",
       " 0.0         0.0         0.0           0.0        0.00415899  0.051703\n",
       " 0.0         0.0406493   0.0487829     0.0282261  0.0718228   0.0\n",
       " 0.00226201  0.06552     0.031723      0.0        0.042321    0.0711757\n",
       " 0.00296154  0.0125568   0.0337406     0.0426124  0.0409554   0.0\n",
       " 0.0         0.0330159   0.0367434  …  0.0        0.0496282   0.0"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1[1:3](sampleImage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 1 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(32, 32, 8, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 2 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(32, 32, 8, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 3 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(16, 16, 8, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 4 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(14, 14, 4, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 5 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(14, 14, 4, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 6 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(7, 7, 4, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 7 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(196, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 8 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(80, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 9 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(80, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 10 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(40, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 11 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(40, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 12 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 13 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#This function just debugs layer by layer.\n",
    "function debugModel(model)\n",
    "    for topLayer = 1:length(model)\n",
    "        print(\"Output after $topLayer layers\"); flush(stdout) #need to flush Standard output because of race between print and display\n",
    "        outputSize = size(model[1:topLayer](sampleImage))\n",
    "        display(outputSize)\n",
    "    end\n",
    "end\n",
    "debugModel(model1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task 1**: Modify the model to have (5x5) convolutions in the first laye and no max pooling layer after the second convolutional layer. You'll need to update the number of neurons in the dense layers. The use `debugModel()` to print the size evolution of your model. Call this model, `model2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 1 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(32, 32, 8, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 2 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(32, 32, 8, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 3 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(16, 16, 8, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 4 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(14, 14, 4, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 5 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(14, 14, 4, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 6 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(7, 7, 4, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 7 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(196, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 8 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(80, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 9 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(80, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 10 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(40, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 11 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(40, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 12 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output after 13 layers"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "using Flux\n",
    "\n",
    "function buildModel2()\n",
    "    Chain(  #Assuming 32x32x3 input layer (like CIFAR10)\n",
    "      Conv((5, 5), 3 => 8, relu, pad=(2, 2), stride=(1, 1)),   #32x32x8 convolutional layer\n",
    "      BatchNorm(8),\n",
    "      x -> maxpool(x, (2, 2)),  #16x16x8\n",
    "      Conv((3, 3), 8 => 4, relu, pad=(0, 0), stride=(1, 1)), #14x14x4  convolutional layer\n",
    "      BatchNorm(4),\n",
    "      x -> maxpool(x, (2, 2)), #7x7x4 ,\n",
    "      flatten,  #196 neurons\n",
    "      Dense(196, 80, relu),\n",
    "      Dropout(0.5),\n",
    "      Dense(80, 40, relu), #40 neurons\n",
    "      Dropout(0.5),\n",
    "      Dense(40, 3), #3 output neruons\n",
    "      softmax)\n",
    "end\n",
    "\n",
    "model2 = buildModel2()\n",
    "debugModel(model2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((x)->x^2)(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32×32×8×1 Array{Float32,4}:\n",
       "[:, :, 1, 1] =\n",
       " 0.0      0.0        0.0         …  0.0        0.0        0.0886668\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0\n",
       " 0.0      0.0        0.0            0.0        0.0487654  0.0705654\n",
       " 0.0      0.0        0.0            0.0101299  0.263558   0.0978056\n",
       " 0.0      0.0        0.0            0.0        0.324143   0.209483\n",
       " 0.0      0.0        0.0         …  0.0        0.144252   0.295569\n",
       " 0.0      0.0        0.0            0.0        0.0        0.175278\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0919172\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0697464\n",
       " 0.0      0.0        0.0            0.0        0.0        0.14019\n",
       " 0.0      0.0        0.0         …  0.0        0.0        0.201373\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0890262\n",
       " 0.0      0.0        0.0            0.0        0.0439698  0.0678641\n",
       " ⋮                               ⋱             ⋮          \n",
       " 0.0      0.0401361  0.0         …  0.0540149  0.0392415  0.0205216\n",
       " 0.0      0.0841348  0.0316437      0.0460354  0.0        0.0\n",
       " 0.0      0.0        0.00950724     0.0605675  0.0        0.0\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0044861\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0\n",
       " 0.0      0.0        0.0         …  0.0        0.0        0.0\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0128444\n",
       " 0.0      0.0        0.0            0.0        0.0        0.0777351\n",
       " 0.0      0.0        0.0            0.0        0.0        0.1\n",
       " 0.0      0.0518492  0.0192885   …  0.0        0.0        0.0\n",
       " 0.12863  0.0997542  0.0816917      0.118012   0.124214   0.0\n",
       "\n",
       "[:, :, 2, 1] =\n",
       " 0.305168  0.195732  0.0334908  0.0159199  …  0.0435569  0.0        0.0\n",
       " 0.366417  0.318569  0.112146   0.133104      0.164785   0.0        0.0\n",
       " 0.310053  0.402621  0.283357   0.248253      0.376024   0.252197   0.0874391\n",
       " 0.289906  0.364041  0.308701   0.224312      0.389265   0.241901   0.128279\n",
       " 0.327482  0.367362  0.244476   0.156333      0.223954   0.149624   0.0276188\n",
       " 0.29077   0.386166  0.194049   0.185746   …  0.229986   0.122617   0.0\n",
       " 0.398737  0.511866  0.229534   0.212693      0.303175   0.0719667  0.0\n",
       " 0.501375  0.618278  0.322904   0.28292       0.301072   0.0499868  0.0\n",
       " 0.468445  0.615647  0.389507   0.354083      0.301654   0.0893276  0.0\n",
       " 0.430033  0.557324  0.407794   0.363469      0.213281   0.12866    0.0\n",
       " 0.399129  0.44474   0.324514   0.209905   …  0.207548   0.19577    0.0\n",
       " 0.401106  0.408677  0.244356   0.15994       0.385648   0.280852   0.0591125\n",
       " 0.368582  0.484134  0.382091   0.250885      0.476319   0.354953   0.160215\n",
       " ⋮                                         ⋱             ⋮          \n",
       " 0.415513  0.626853  0.537607   0.542146   …  0.0248968  0.0350337  0.0\n",
       " 0.241227  0.429884  0.453108   0.422627      0.0170952  0.017188   0.0\n",
       " 0.230335  0.28556   0.285061   0.311952      0.0321684  0.0123985  0.0\n",
       " 0.227553  0.256039  0.183923   0.205258      0.0190786  0.0        0.0\n",
       " 0.219949  0.282653  0.233116   0.199217      0.0        0.0        0.0\n",
       " 0.233482  0.280719  0.232249   0.179366   …  0.18363    0.0687761  0.0\n",
       " 0.133796  0.19168   0.144568   0.135701      0.226209   0.112926   0.0\n",
       " 0.171973  0.218598  0.154151   0.181278      0.276763   0.254312   0.0859837\n",
       " 0.210817  0.21218   0.093715   0.126439      0.0932478  0.157302   0.0483952\n",
       " 0.216479  0.238239  0.160354   0.170866      0.0566772  0.159699   0.0162033\n",
       " 0.16875   0.185748  0.117501   0.106145   …  0.0        0.111978   0.114637\n",
       " 0.252424  0.307256  0.331001   0.337162      0.384333   0.403848   0.32813\n",
       "\n",
       "[:, :, 3, 1] =\n",
       " 0.116867     0.0  0.0  0.0  0.0        …  0.0        0.0        0.0\n",
       " 0.047676     0.0  0.0  0.0  0.0           0.0        0.0        0.0529605\n",
       " 0.00156586   0.0  0.0  0.0  0.0           0.0        0.0        0.0\n",
       " 0.0          0.0  0.0  0.0  0.0           0.0        0.0        0.0330889\n",
       " 0.00842832   0.0  0.0  0.0  0.0           0.0        0.0        0.0501923\n",
       " 0.0407197    0.0  0.0  0.0  0.0        …  0.0        0.0        0.0315689\n",
       " 0.0288357    0.0  0.0  0.0  0.0           0.0        0.0        0.0222085\n",
       " 0.037257     0.0  0.0  0.0  0.0           0.0        0.0        0.0\n",
       " 0.0738307    0.0  0.0  0.0  0.0           0.0        0.0        0.0\n",
       " 0.073293     0.0  0.0  0.0  0.0           0.0        0.0        0.0\n",
       " 0.0555855    0.0  0.0  0.0  0.0        …  0.0        0.0        0.0\n",
       " 0.0181989    0.0  0.0  0.0  0.0           0.0        0.0        0.0\n",
       " 0.0195624    0.0  0.0  0.0  0.0434071     0.0        0.0        0.0\n",
       " ⋮                                      ⋱             ⋮          \n",
       " 0.0          0.0  0.0  0.0  0.0        …  0.0        0.0        0.0169833\n",
       " 0.0          0.0  0.0  0.0  0.0           0.0        0.0        0.0580561\n",
       " 0.00135208   0.0  0.0  0.0  0.0           0.0        0.0        0.0785408\n",
       " 0.0          0.0  0.0  0.0  0.0           0.0428123  0.0182991  0.12537\n",
       " 0.0          0.0  0.0  0.0  0.0           0.0        0.0        0.158327\n",
       " 0.0          0.0  0.0  0.0  0.0        …  0.0        0.0        0.0745299\n",
       " 0.0142163    0.0  0.0  0.0  0.0           0.0        0.0        0.0892123\n",
       " 0.0286224    0.0  0.0  0.0  0.0           0.0        0.0        0.0589481\n",
       " 0.00819617   0.0  0.0  0.0  0.0           0.0        0.0        0.0241281\n",
       " 0.000536534  0.0  0.0  0.0  0.0           0.0        0.0        0.0\n",
       " 0.0          0.0  0.0  0.0  0.0        …  0.0        0.0        0.0\n",
       " 0.0          0.0  0.0  0.0  0.0           0.0        0.0        0.0\n",
       "\n",
       "[:, :, 4, 1] =\n",
       " 0.0  0.0        0.0280093  0.00326391  …  0.00171085  0.239608    0.110939\n",
       " 0.0  0.0742084  0.0        0.0            0.043286    0.202498    0.106062\n",
       " 0.0  0.081248   0.106085   0.132868       0.164123    0.376589    0.476379\n",
       " 0.0  0.0211832  0.0499174  0.169455       0.236195    0.327162    0.447089\n",
       " 0.0  0.0        0.0923426  0.200863       0.219736    0.249209    0.458505\n",
       " 0.0  0.0        0.167296   0.123299    …  0.21295     0.256689    0.34704\n",
       " 0.0  0.0691045  0.156699   0.126869       0.240146    0.376399    0.221485\n",
       " 0.0  0.109778   0.147914   0.171636       0.218605    0.474407    0.214093\n",
       " 0.0  0.0871027  0.144909   0.243482       0.191553    0.473496    0.255626\n",
       " 0.0  0.0550436  0.121788   0.278056       0.224038    0.402914    0.361969\n",
       " 0.0  0.0        0.0660906  0.302321    …  0.208315    0.359303    0.455475\n",
       " 0.0  0.0        0.106703   0.370031       0.259463    0.400711    0.43731\n",
       " 0.0  0.0        0.263207   0.384565       0.284593    0.430769    0.438995\n",
       " ⋮                                      ⋱              ⋮           \n",
       " 0.0  0.0899136  0.297032   0.227189    …  0.144765    0.214986    0.145124\n",
       " 0.0  0.0383542  0.200922   0.244018       0.116119    0.183802    0.0909234\n",
       " 0.0  0.0        0.0554151  0.173081       0.146007    0.18287     0.0800369\n",
       " 0.0  0.0328197  0.0585888  0.0746339      0.114468    0.162416    0.0789013\n",
       " 0.0  0.0696554  0.0895758  0.0702159      0.0891017   0.10372     0.052508\n",
       " 0.0  0.0394809  0.0614714  0.0750011   …  0.0807883   0.141754    0.150723\n",
       " 0.0  0.0        0.0592015  0.0636279      0.0761919   0.0858762   0.0987167\n",
       " 0.0  0.0        0.0776786  0.0595474      0.103903    0.20372     0.234776\n",
       " 0.0  0.042587   0.0482366  0.0155949      0.0         0.00648458  0.243365\n",
       " 0.0  0.0441965  0.0279279  0.0472243      0.00544468  0.0816212   0.2912\n",
       " 0.0  0.114941   0.0884062  0.0791137   …  0.0         0.0311805   0.32432\n",
       " 0.0  0.129237   0.141781   0.142687       0.104532    0.142915    0.406182\n",
       "\n",
       "[:, :, 5, 1] =\n",
       " 0.42036   0.643609  0.411076  0.396068   …  0.592231    0.632326   0.237503\n",
       " 0.359208  0.52674   0.236607  0.2434        0.363167    0.439521   0.0139677\n",
       " 0.4402    0.604437  0.381234  0.437097      0.594214    0.363888   0.0\n",
       " 0.496032  0.655685  0.458559  0.588399      0.66686     0.226214   0.0\n",
       " 0.568165  0.735989  0.503922  0.58716       0.644286    0.137469   0.0\n",
       " 0.650064  0.779706  0.613689  0.641749   …  0.61394     0.171698   0.0\n",
       " 0.597148  0.811022  0.736419  0.639205      0.49081     0.299095   0.0\n",
       " 0.499187  0.784265  0.664472  0.529023      0.372849    0.358883   0.0621536\n",
       " 0.420489  0.727281  0.607285  0.480365      0.40733     0.398339   0.0593184\n",
       " 0.391483  0.793005  0.686908  0.497118      0.49731     0.388018   0.0259491\n",
       " 0.459936  0.911881  0.708398  0.435886   …  0.491702    0.343428   0.0232605\n",
       " 0.596736  1.00576   0.689496  0.336976      0.419663    0.268526   0.0\n",
       " 0.717308  1.05117   0.62472   0.294019      0.274374    0.111598   0.0\n",
       " ⋮                                        ⋱              ⋮          \n",
       " 0.540262  0.548284  0.400683  0.440651   …  0.0915601   0.0        0.0\n",
       " 0.501802  0.506274  0.341745  0.403814      0.00266102  0.0        0.0\n",
       " 0.422165  0.506269  0.32057   0.362008      0.0         0.0111199  0.0\n",
       " 0.320288  0.444858  0.333507  0.311065      0.0357709   0.0607408  0.0\n",
       " 0.291385  0.376832  0.318707  0.333543      0.147961    0.250241   0.123934\n",
       " 0.286761  0.376281  0.264283  0.238119   …  0.18443     0.26382    0.108189\n",
       " 0.333395  0.386593  0.251841  0.251957      0.0821197   0.272334   0.12512\n",
       " 0.366273  0.421309  0.30157   0.271006      0.062764    0.262318   0.194355\n",
       " 0.346799  0.424248  0.259814  0.222185      0.243514    0.41308    0.216035\n",
       " 0.336999  0.401394  0.210622  0.19009       0.290658    0.41372    0.180596\n",
       " 0.262074  0.272937  0.183729  0.208625   …  0.364936    0.369107   0.202185\n",
       " 0.145001  0.129254  0.052779  0.0813856     0.179986    0.0516552  0.0\n",
       "\n",
       "[:, :, 6, 1] =\n",
       " 0.0        0.0         0.0        …  0.0         0.0          0.0  0.0\n",
       " 0.209002   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.196752   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.129451   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.102661   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.138727   0.0         0.0        …  0.0         0.0          0.0  0.0\n",
       " 0.228012   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.336949   0.0         0.0           0.0316691   0.0          0.0  0.0\n",
       " 0.352244   0.0         0.0           0.00165671  0.0          0.0  0.0\n",
       " 0.309072   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.286386   0.0200806   0.0        …  0.0         0.0          0.0  0.0\n",
       " 0.304184   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.264775   0.0         0.0           0.115974    0.0          0.0  0.0\n",
       " ⋮                                 ⋱                           ⋮    \n",
       " 0.42568    0.1044      0.0        …  0.0         0.0          0.0  0.0\n",
       " 0.257611   0.0577773   0.0           0.0         0.0          0.0  0.0\n",
       " 0.165305   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.154149   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.154066   0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.11787    0.0         0.0        …  0.0         0.0          0.0  0.0\n",
       " 0.0584135  0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.0252469  0.0         0.0           0.0         0.0508483    0.0  0.0\n",
       " 0.0646552  0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.0785731  0.0         0.0           0.0         0.0          0.0  0.0\n",
       " 0.192222   0.0         0.0        …  0.0         0.000849967  0.0  0.0\n",
       " 0.19234    0.00187336  0.0526274     0.0343262   0.209961     0.0  0.0\n",
       "\n",
       "[:, :, 7, 1] =\n",
       " 0.0        0.0182837   0.0        …  0.0        0.0          0.0\n",
       " 0.192913   0.288121    0.0           0.0        0.0          0.0\n",
       " 0.35519    0.280802    0.0917159     0.0619874  0.0          0.0\n",
       " 0.282015   0.216222    0.0           0.150853   0.0          0.0\n",
       " 0.348145   0.239349    0.0           0.171153   0.0          0.0\n",
       " 0.313251   0.242423    0.0        …  0.180583   0.0          0.0\n",
       " 0.234546   0.268487    0.0           0.163284   0.0          0.0\n",
       " 0.285873   0.322851    0.107351      0.0        0.0          0.0\n",
       " 0.322759   0.354368    0.198814      0.0        0.0          0.0\n",
       " 0.294394   0.365203    0.229769      0.0        0.0          0.0\n",
       " 0.246179   0.35286     0.140347   …  0.0        0.0          0.0\n",
       " 0.253233   0.317643    0.0           0.0669491  0.0255055    0.0\n",
       " 0.324411   0.400339    0.16111       0.201702   0.167944     0.0\n",
       " ⋮                                 ⋱             ⋮            \n",
       " 0.298216   0.305311    0.147474   …  0.0        0.0          0.0\n",
       " 0.238791   0.303844    0.11487       0.0        0.0          0.0\n",
       " 0.155952   0.212812    0.0880894     0.0        0.0          0.0\n",
       " 0.121509   0.127216    0.016405      0.0        0.0          0.0\n",
       " 0.0819063  0.115883    0.0386947     0.0        0.0          0.0\n",
       " 0.180305   0.16089     0.025595   …  0.0        0.000940172  0.0\n",
       " 0.160209   0.0914977   0.0           0.0480096  0.129121     0.0\n",
       " 0.0886761  0.0655931   0.0           0.0432364  0.157173     0.0\n",
       " 0.0811302  0.0828747   0.0           0.0        0.111303     0.0\n",
       " 0.113775   0.111691    0.0           0.0        0.0502554    0.0\n",
       " 0.0        0.00693878  0.0        …  0.0        0.0          0.0\n",
       " 0.0693289  0.1148      0.0957809     0.103044   0.194711     0.0573601\n",
       "\n",
       "[:, :, 8, 1] =\n",
       " 0.0794301  0.112223    0.0         …  0.0349633  0.0       0.0\n",
       " 0.149134   0.196175    0.237904       0.177356   0.0       0.083229\n",
       " 0.144219   0.0         0.0            0.0        0.0       0.0307051\n",
       " 0.100436   0.0         0.0            0.0        0.0       0.0543363\n",
       " 0.126676   0.107159    0.0            0.0        0.0       0.0\n",
       " 0.101436   0.120129    0.0         …  0.0        0.0       0.0\n",
       " 0.141096   0.153279    0.0            0.0        0.0       0.0\n",
       " 0.264559   0.162372    0.0            0.0        0.0       0.0\n",
       " 0.271048   0.0596872   0.0            0.0        0.0       0.0\n",
       " 0.234877   0.0         0.0            0.0        0.0       0.0\n",
       " 0.183334   0.0         0.0         …  0.0        0.0       0.00924824\n",
       " 0.0969054  0.0         0.0            0.0        0.115333  0.0947739\n",
       " 0.0772381  0.0         0.0158903      0.0        0.170414  0.0494857\n",
       " ⋮                                  ⋱             ⋮         \n",
       " 0.129612   0.141781    0.0247526   …  0.0        0.0       0.0\n",
       " 0.0347737  0.0462628   0.0            0.0346092  0.0       0.0\n",
       " 0.0106446  0.0         0.0108581      0.0183988  0.0       0.0\n",
       " 0.0664177  0.0         0.0            0.0274466  0.0       0.0\n",
       " 0.0556898  0.0111066   0.0            0.0        0.0       0.00662558\n",
       " 0.0807402  0.0169031   0.0         …  0.111505   0.0       0.13677\n",
       " 0.0148074  0.0         0.0            0.113176   0.0       0.173027\n",
       " 0.0        0.0013035   0.0            0.0        0.0       0.0797729\n",
       " 0.0402551  0.0124161   0.0            0.0        0.0       0.0928724\n",
       " 0.0446077  0.0167852   0.0            0.0        0.0       0.0991374\n",
       " 0.0        0.00905571  0.00857221  …  0.0        0.0       0.167879\n",
       " 0.112149   0.124697    0.131626       0.151799   0.222899  0.30889"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Conv((5, 5), 3 => 8, relu, pad=(2, 2), stride=(1, 1))(sampleImage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Float32[0.37642843 0.5598695 … 0.5232896 0.51115525; 0.23616588 0.42534563 … 0.30818358 0.36634704; … ; 0.0 0.092745684 … 0.029186321 0.286302; 0.0 0.0 … 0.028379135 0.18822518]\n",
       "\n",
       "Float32[0.22482051 0.20179287 … 0.0593405 0.0; 0.21052463 0.09745899 … 0.0 0.0; … ; 0.03943745 0.0 … 0.0 0.0; 0.011902696 0.0 … 0.0 0.0]\n",
       "\n",
       "Float32[0.11962601 0.0 … 0.0 0.0; 0.101407245 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]\n",
       "\n",
       "Float32[0.0 0.0 … 0.28450823 0.2550626; 0.0 0.037507117 … 0.47264296 0.37121248; … ; 0.034477532 0.05805671 … 0.43471462 0.3371518; 0.15588157 0.14918739 … 0.5822751 0.37097153]\n",
       "\n",
       "Float32[0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]\n",
       "\n",
       "Float32[0.25806904 0.15887585 … 0.10860255 0.0; 0.20317243 0.12383165 … 0.0 0.0; … ; 0.1334772 0.0 … 0.07891907 0.0; 0.13053887 0.0 … 0.14456542 0.0]\n",
       "\n",
       "Float32[0.27703243 0.22442995 … 0.0 0.0; 0.33960533 0.24779922 … 0.0 0.0; … ; 0.2719728 0.22460933 … 0.15617509 0.014591235; 0.22359945 0.18527597 … 0.23008047 0.07049643]\n",
       "\n",
       "Float32[0.09005112 0.11411203 … 0.009999712 0.019283753; 0.12789842 0.16116221 … 0.07657484 0.10325146; … ; 0.0 0.0 … 0.013852818 0.06125663; 0.0 0.0 … 0.11986 0.16948956], Float32[0.011576572 0.55298305 … 0.6408517 0.49298158; 0.0 0.0 … 0.057297815 0.261568; … ; 0.0 0.0043642083 … 0.0 0.16655631; 0.0 0.0 … 0.0 0.0573189]\n",
       "\n",
       "Float32[0.0 0.0 … 0.029344529 0.1911893; 0.0 0.0 … 0.0 0.29311728; … ; 0.0 0.007262538 … 0.0 0.16661716; 0.0 0.0 … 0.0 0.19188401]\n",
       "\n",
       "Float32[0.1682048 0.06297117 … 0.08534347 0.30364963; 0.22679697 0.32247332 … 0.28572136 0.47471943; … ; 0.17704381 0.14159691 … 0.16958436 0.40919977; 0.12357393 0.12333427 … 0.21309683 0.23047504]\n",
       "\n",
       "Float32[0.0 0.059996374 … 0.11861298 0.25821254; 0.0 0.14649323 … 0.2549305 0.1745407; … ; 0.0 0.17495598 … 0.094693676 0.017602943; 0.035000987 0.14092909 … 0.1620394 0.0]\n",
       "\n",
       "Float32[0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]\n",
       "\n",
       "Float32[0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.030237118 0.0 … 0.0 0.0]\n",
       "\n",
       "Float32[0.7754154 0.8297039 … 0.8014235 0.29657644; 0.8460343 0.9330562 … 1.0230908 0.4885396; … ; 0.47232074 0.5090967 … 0.838753 0.447139; 0.32868856 0.35272154 … 0.6174536 0.40635085]\n",
       "\n",
       "Float32[0.05373172 0.12599465 … 0.0910154 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Flux\n",
    "\n",
    "function buildBorkenModel3()\n",
    "    Chain(  #Assuming 32x32x3 input layer (like CIFAR10)\n",
    "      (prev)->(\n",
    "            (Conv((5, 5), 3 => 8, relu, pad=(2, 2), stride=(1, 1))(prev),   #32x32x8 convolutional layer\n",
    "             Conv((3, 3), 3 => 8, relu, pad=(1, 1), stride=(1, 1))(prev))\n",
    "                )\n",
    "        )\n",
    "#       BatchNorm(8),\n",
    "#       x -> maxpool(x, (2, 2)),  #16x16x8\n",
    "#       Conv((3, 3), 8 => 4, relu, pad=(0, 0), stride=(1, 1)), #14x14x4  convolutional layer\n",
    "#       BatchNorm(4),\n",
    "#       x -> maxpool(x, (2, 2)), #7x7x4 ,\n",
    "#       flatten,  #196 neurons\n",
    "#       Dense(196, 80, relu),\n",
    "#       Dropout(0.5),\n",
    "#       Dense(80, 40, relu), #40 neurons\n",
    "#       Dropout(0.5),\n",
    "#       Dense(40, 3), #3 output neruons\n",
    "#       softmax)\n",
    "end\n",
    "\n",
    "model3 = buildBorkenModel3()\n",
    "model3(sampleImage)\n",
    "# debugModel(model3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now train the model using basic functions of `Flux.jl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "search:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/latex": [
       "\\begin{verbatim}\n",
       "crossentropy(ŷ, y; dims=1, ϵ=eps(ŷ), agg=mean)\n",
       "\\end{verbatim}\n",
       "Return the cross entropy between the given probability distributions; calculated as\n",
       "\n",
       "\\begin{verbatim}\n",
       "agg(-sum(y .* log.(ŷ .+ ϵ); dims=dims))\n",
       "\\end{verbatim}\n",
       "Cross entropy is tipically used as a loss in multi-class classification, in which case the labels \\texttt{y} are given in a one-hot format.  \\texttt{dims} specifies the dimension (or the dimensions) containing the class probabilities. The prediction \\texttt{ŷ} is supposed to sum to one across \\texttt{dims}, as would be the case with the output of a \\href{@ref}{\\texttt{softmax}} operation. \n",
       "\n",
       "Use of \\href{@ref}{\\texttt{logitcrossentropy}} is recomended over \\texttt{crossentropy} for  numerical stability.\n",
       "\n",
       "See also: \\href{@ref}{\\texttt{Flux.logitcrossentropy}}, \\href{@ref}{\\texttt{Flux.binarycrossentropy}}, \\href{@ref}{\\texttt{Flux.logitbinarycrossentropy}}\n",
       "\n"
      ],
      "text/markdown": [
       "```\n",
       "crossentropy(ŷ, y; dims=1, ϵ=eps(ŷ), agg=mean)\n",
       "```\n",
       "\n",
       "Return the cross entropy between the given probability distributions; calculated as\n",
       "\n",
       "```\n",
       "agg(-sum(y .* log.(ŷ .+ ϵ); dims=dims))\n",
       "```\n",
       "\n",
       "Cross entropy is tipically used as a loss in multi-class classification, in which case the labels `y` are given in a one-hot format.  `dims` specifies the dimension (or the dimensions) containing the class probabilities. The prediction `ŷ` is supposed to sum to one across `dims`, as would be the case with the output of a [`softmax`](@ref) operation. \n",
       "\n",
       "Use of [`logitcrossentropy`](@ref) is recomended over `crossentropy` for  numerical stability.\n",
       "\n",
       "See also: [`Flux.logitcrossentropy`](@ref), [`Flux.binarycrossentropy`](@ref), [`Flux.logitbinarycrossentropy`](@ref)\n"
      ],
      "text/plain": [
       "\u001b[36m  crossentropy(ŷ, y; dims=1, ϵ=eps(ŷ), agg=mean)\u001b[39m\n",
       "\n",
       "  Return the cross entropy between the given probability distributions;\n",
       "  calculated as\n",
       "\n",
       "\u001b[36m  agg(-sum(y .* log.(ŷ .+ ϵ); dims=dims))\u001b[39m\n",
       "\n",
       "  Cross entropy is tipically used as a loss in multi-class classification, in\n",
       "  which case the labels \u001b[36my\u001b[39m are given in a one-hot format. \u001b[36mdims\u001b[39m specifies the\n",
       "  dimension (or the dimensions) containing the class probabilities. The\n",
       "  prediction \u001b[36mŷ\u001b[39m is supposed to sum to one across \u001b[36mdims\u001b[39m, as would be the case\n",
       "  with the output of a \u001b[36msoftmax\u001b[39m operation. \n",
       "\n",
       "  Use of \u001b[36mlogitcrossentropy\u001b[39m is recomended over \u001b[36mcrossentropy\u001b[39m for numerical\n",
       "  stability.\n",
       "\n",
       "  See also: \u001b[36mFlux.logitcrossentropy\u001b[39m, \u001b[36mFlux.binarycrossentropy\u001b[39m,\n",
       "  \u001b[36mFlux.logitbinarycrossentropy\u001b[39m"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "? crossentropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "loss (generic function with 1 method)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#define the loss. function\n",
    "using Flux\n",
    "using Flux: crossentropy\n",
    "loss(x, y, model) = crossentropy(model(x), y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.089335f0"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss(x_train_batches[1],y_train_batches[1],model1) #Loss on first batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.100507f0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss(x_train,y_train,model1) #Loss on all the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "accuracy (generic function with 1 method)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#define the accuracy function\n",
    "using Flux: onecold\n",
    "using Statistics\n",
    "accuracy(x, y, model) = mean(onecold(model(x)) .== onecold(y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.338"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(x_validate,y_validate,model1) #should be around 0.33 as model is garbage at this point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Params([Float32[-0.05195587 -0.1914966 0.1993872; 0.03160675 -0.14784676 -0.1522569; -0.13879706 0.14692861 0.014990601]\n",
       "\n",
       "Float32[0.14910218 0.23134007 -0.15734461; 0.23312914 -0.2116273 0.061777767; 0.100441955 -0.22437473 -0.23886418]\n",
       "\n",
       "Float32[0.15397471 0.07025016 -0.029751822; 0.07348987 0.13914828 -0.24556494; 0.032805998 -0.22804672 -0.23153886]\n",
       "\n",
       "Float32[-0.0450734 0.016496588 -0.18961361; 0.24571954 0.022647547 -0.20804858; -0.19531497 -0.1843612 -0.18160343]\n",
       "\n",
       "Float32[-0.21475421 0.19068167 0.13945103; 0.14721744 0.23211372 0.06377409; -0.1704689 -0.14928871 0.1008375]\n",
       "\n",
       "Float32[-0.1352752 0.017503316 -0.14445105; 0.10093434 -0.02187442 0.1893007; -0.051454913 -0.13178863 0.20612815]\n",
       "\n",
       "Float32[-0.24083011 -0.072478965 -0.23012586; -0.11381963 -0.10176159 -0.0016638745; -0.22292168 0.10561025 0.06790184]\n",
       "\n",
       "Float32[-0.18939033 0.19719625 -0.16786315; 0.035674814 0.22406305 -0.068515554; 0.16462891 0.22718161 0.03286369]\n",
       "\n",
       "Float32[-0.04848667 -0.12963219 -0.087829486; -0.049651932 -0.010685997 0.17585108; 0.10060008 -0.21450746 0.22568983]\n",
       "\n",
       "Float32[-0.09029172 0.1236272 -0.20196013; 0.169513 -0.08597045 0.17330256; 0.05183367 0.21291372 -0.12686221]\n",
       "\n",
       "Float32[0.22592449 -0.11720096 -0.07725494; -0.08033095 -0.027259123 -0.21586993; 0.09688606 -0.108145386 -0.1251697]\n",
       "\n",
       "Float32[-0.09387291 0.12741546 -0.23784143; -0.18188271 0.22716348 -0.23464082; -0.11664489 -0.115675196 -0.14265934]\n",
       "\n",
       "Float32[-0.04515751 -0.18567203 0.22952025; -0.13498901 -0.03819915 -0.15313134; 0.21625309 0.01352846 0.07580073]\n",
       "\n",
       "Float32[0.06847823 -0.20115067 0.02004497; 0.045748096 -0.24100941 -0.10688169; 0.07083088 0.13168731 -0.034791693]\n",
       "\n",
       "Float32[-0.24041912 0.061162647 -0.06471391; 0.003667767 -0.20042463 0.23459287; 0.028224766 0.012035387 -0.23487812]\n",
       "\n",
       "Float32[0.14811318 0.17207178 -0.054914314; -0.18180019 -0.12880893 0.045385715; 0.22928189 0.018855698 0.15660986]\n",
       "\n",
       "Float32[0.17540705 -0.028145527 0.08958656; -0.1275649 -0.05949484 -0.19867928; 0.047810625 0.1766522 -0.10339623]\n",
       "\n",
       "Float32[-0.08192498 0.18364373 -0.04299814; -0.21290462 -0.043569528 0.19584844; 0.19410633 -0.12193727 0.11066538]\n",
       "\n",
       "Float32[0.20822684 -0.124182925 -0.03759084; 0.1461151 0.12038092 -0.018718882; -0.020758284 0.15199372 -0.08696104]\n",
       "\n",
       "Float32[-0.08253558 -0.108488634 0.12141835; -0.103511564 0.101979405 -0.041506354; -0.18311225 -0.022688633 -0.006599621]\n",
       "\n",
       "Float32[-0.018733261 -0.014213425 -0.10747151; -0.062586635 -0.14669143 0.14684667; 0.15177268 0.035950266 -0.19956863]\n",
       "\n",
       "Float32[-0.1325881 0.14570582 -0.009933944; -0.15325212 0.13839072 0.027392946; 0.10823953 -0.077405676 -0.06360246]\n",
       "\n",
       "Float32[-0.040526215 0.13385409 0.08596616; 0.08420621 -0.067130364 -0.043241486; -0.112472706 0.20132107 -0.21916576]\n",
       "\n",
       "Float32[0.007092538 0.058198396 -0.20224592; -0.2393372 -0.21999975 0.049352355; 0.18611172 0.16603352 -0.19718568], Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], Float32[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], Float32[-0.08283963 0.019276788 -0.06651807; -0.20015235 0.01857142 -0.10584939; 0.14539792 -0.099053636 -0.14711171]\n",
       "\n",
       "Float32[0.014715825 -0.17135954 -0.06786924; -0.036092594 0.023404427 0.16372511; -0.12144277 -0.0392735 0.11346527]\n",
       "\n",
       "Float32[0.08387386 0.008632293 0.17859705; 0.22960389 0.121690705 -0.09651291; -0.22750531 0.034796212 0.18695442]\n",
       "\n",
       "Float32[0.2224436 0.092617646 -0.20502217; -0.21909645 -0.21059202 -0.11618487; 0.19851217 0.0113571705 0.14786284]\n",
       "\n",
       "Float32[0.123022154 -0.020622622 -0.08577524; -0.19381762 0.21823548 0.041791465; 0.1543679 0.023010943 -0.013613994]\n",
       "\n",
       "Float32[-0.0023398807 0.19264375 -0.20714238; -0.08989131 -0.11309118 0.10807323; 0.051407468 -0.006629306 -0.1254946]\n",
       "\n",
       "Float32[0.20309436 0.07133821 0.0306434; 0.090061575 0.21047457 0.09303478; -0.17627403 0.16560593 -0.15188415]\n",
       "\n",
       "Float32[-0.16341986 -0.11627383 0.06766087; 0.046139505 -0.105202466 0.20914255; 0.20641807 0.22008365 -0.12865123]\n",
       "\n",
       "Float32[-0.15327837 -0.2056741 0.21994838; -0.09937918 0.083060704 -0.025723683; -0.064529575 -0.1853596 0.22263213]\n",
       "\n",
       "Float32[0.022027742 0.026474008 -0.05506435; -0.108780116 0.20904899 -0.13029552; 0.015632603 0.04455765 0.019398509]\n",
       "\n",
       "Float32[-0.027531445 -0.06904396 0.027223099; 0.025935821 -0.07688597 0.18255346; 0.024857257 -0.12527286 -0.20791855]\n",
       "\n",
       "Float32[0.11603983 0.0755131 0.21239972; -0.2208932 0.02721377 -0.2198406; -0.19640554 -0.009417742 0.13560715]\n",
       "\n",
       "Float32[-0.14280453 0.18293548 0.15559424; -0.004483694 0.035585202 0.10128275; -0.020652292 0.1883492 0.13394251]\n",
       "\n",
       "Float32[-0.1403363 -0.16108648 0.18757196; -0.15675722 -0.021354178 0.07419357; -0.03254147 -0.14220454 0.14694397]\n",
       "\n",
       "Float32[-0.06937877 -0.047864042 0.08752181; -0.02879883 0.054063167 0.059277296; -0.22179735 -0.034253925 -0.19651188]\n",
       "\n",
       "Float32[0.18081117 -0.13451897 0.009022742; 0.17408009 0.1431521 -0.05686711; -0.145961 -0.07239519 0.11386089]\n",
       "\n",
       "Float32[-0.098292634 0.083980404 0.16524504; -0.1897255 -0.16222395 0.1985497; -0.18278499 0.1474925 0.1289289]\n",
       "\n",
       "Float32[0.02611031 0.13947955 -0.07815099; 0.072402894 0.16029745 0.12125626; -0.049335133 0.022760646 -0.019579966]\n",
       "\n",
       "Float32[-0.09516545 0.011618594 0.09166878; -0.12263221 0.15234715 0.030982036; -0.0586252 -0.004076106 -0.057898417]\n",
       "\n",
       "Float32[-0.22253402 -0.22605659 -0.017846268; 0.029130273 -0.06598701 -0.099170916; 0.11905434 -0.0145084625 0.15849541]\n",
       "\n",
       "Float32[0.2026861 0.11215636 -0.093385056; -0.17716439 -0.12924461 0.14943199; 0.18310794 -0.054793153 -0.21380652]\n",
       "\n",
       "Float32[-0.014665867 -0.081042774 0.010872145; 0.18516369 0.06895893 0.011568354; 0.11416412 0.22093463 -0.18851481]\n",
       "\n",
       "Float32[0.10848897 -0.04469589 -0.056479137; -0.17728285 -0.21064647 0.116500355; 0.20796593 -0.18933882 -0.17221078]\n",
       "\n",
       "Float32[-0.20819385 0.12634034 -0.07358682; -0.10709818 0.050128844 -0.07897707; 0.12983629 0.15389927 -0.17205243]\n",
       "\n",
       "Float32[-0.0826888 0.029848736 -0.1698876; 0.22223477 0.17136818 -0.18017739; -0.07190955 0.036449887 -0.044803]\n",
       "\n",
       "Float32[-0.04364368 0.020305283 -0.19106363; -0.019581707 0.089619204 0.07206802; 0.19423246 0.16722819 -0.09631364]\n",
       "\n",
       "Float32[-0.09488402 0.20554411 0.16634968; 0.05316617 -0.08876244 0.20155449; 0.09154318 -0.17132194 -0.12257753]\n",
       "\n",
       "Float32[0.07744349 -0.022329343 0.06572211; 0.07162981 -0.1548487 -0.14817342; -0.008792788 0.06934657 -0.07023452]\n",
       "\n",
       "Float32[0.16095595 0.010006111 0.04308841; 0.16454445 -0.113114215 0.13284422; -0.1390265 0.093865976 0.07424926]\n",
       "\n",
       "Float32[0.041264124 -0.16432011 0.023263711; 0.010240111 0.06893258 0.19363487; -0.06352991 -0.12086379 -0.18995337]\n",
       "\n",
       "Float32[-0.10461163 -0.07244711 -0.008157157; 0.21977828 0.21197516 0.16861387; 0.004607943 -0.20275012 -0.13645256]\n",
       "\n",
       "Float32[-0.17673843 0.099440545 0.22782198; -0.0023471299 -0.12164795 -0.14320336; -0.06124454 -0.2183468 -0.13822643], Float32[0.0, 0.0, 0.0, 0.0], Float32[0.0, 0.0, 0.0, 0.0], Float32[1.0, 1.0, 1.0, 1.0], Float32[0.021024846 -0.058098745 … -0.11529009 -0.09991467; -0.12891486 0.12644055 … 0.048037775 0.06075859; … ; 0.0011859534 0.09600598 … -0.11861201 0.077586636; -0.13949417 -0.10211134 … -0.0019351672 0.09089559], Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0  …  0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], Float32[-0.054772817 -0.21192645 … -0.20418753 0.19386242; 0.08050962 -0.05890098 … -0.0149200065 -0.12616833; … ; -0.055093702 0.10020686 … 0.12811266 0.1346704; 0.16195284 0.05638044 … -0.2166935 -0.14171837], Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0  …  0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], Float32[-0.29854202 -0.36344635 … -0.027695714 -0.33870307; 0.19871342 0.3024111 … 0.36325443 0.24938005; 0.2774637 -0.29408333 … -0.12915917 -0.32214552], Float32[0.0, 0.0, 0.0]])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params(model1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: Slow fallback implementation invoked for ∇conv_filter!  You probably don't want this; check your datatypes.\n",
      "│   yT = Float32\n",
      "│   T1 = FixedPointNumbers.Normed{UInt8,8}\n",
      "│   T2 = Float32\n",
      "└ @ NNlib /Users/uqjnazar/.julia/packages/NNlib/1WXEb/src/conv.jl:206\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Grads(...)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#example of computing the gradient (automatic differentiantion) on the first minibatch\n",
    "gs = gradient(()->loss(x_train_batches[1],y_train_batches[1],model1),params(model1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example of a single update step of the optimizer\n",
    "using Flux: update!\n",
    "η = 0.002\n",
    "opt = ADAM(η)\n",
    "update!(opt,params(model1),gs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A training loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we use the above for a simple training loop. On each epoch we print:\n",
    "* The epoch number\n",
    "* The wall time\n",
    "* The accuracy computed on the validation set\n",
    "* The loss (computed on the training set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "trainModel (generic function with 1 method)"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Dates\n",
    "function trainModel(model; epochs = 20, η = 0.005)\n",
    "    opt = ADAM(η)\n",
    "    for ep in 1:epochs #Loop over epochs\n",
    "        \n",
    "        for bi in 1:numMiniBatches #Loop over minibatches\n",
    "            gs = gradient(()->loss(x_train_batches[bi],y_train_batches[bi],model),params(model))\n",
    "            update!(opt,params(model),gs)\n",
    "        end\n",
    "        \n",
    "        acc = accuracy(x_validate,y_validate,model)\n",
    "        ls = loss(x_train,y_train,model)\n",
    "        time = Dates.format(now(), \"HH:MM:SS\")\n",
    "        @show ep, time, acc, ls\n",
    "    end\n",
    "    \n",
    "    return model\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This now trains a model of type \"model 1\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(ep, time, acc, ls) = (1, \"15:43:43\", 0.65, 0.7546195f0)\n",
      "(ep, time, acc, ls) = (2, \"15:43:46\", 0.684, 0.66738147f0)\n",
      "(ep, time, acc, ls) = (3, \"15:43:49\", 0.717, 0.60654163f0)\n",
      "(ep, time, acc, ls) = (4, \"15:43:52\", 0.725, 0.5442484f0)\n",
      "(ep, time, acc, ls) = (5, \"15:43:55\", 0.751, 0.47823748f0)\n",
      "(ep, time, acc, ls) = (6, \"15:43:58\", 0.777, 0.42364672f0)\n",
      "(ep, time, acc, ls) = (7, \"15:44:01\", 0.791, 0.41671857f0)\n",
      "(ep, time, acc, ls) = (8, \"15:44:04\", 0.734, 0.50269175f0)\n",
      "(ep, time, acc, ls) = (9, \"15:44:08\", 0.784, 0.34779125f0)\n",
      "(ep, time, acc, ls) = (10, \"15:44:12\", 0.784, 0.3501305f0)\n",
      "(ep, time, acc, ls) = (11, \"15:44:15\", 0.774, 0.37157777f0)\n",
      "(ep, time, acc, ls) = (12, \"15:44:18\", 0.796, 0.31285048f0)\n",
      "(ep, time, acc, ls) = (13, \"15:44:21\", 0.768, 0.33433282f0)\n",
      "(ep, time, acc, ls) = (14, \"15:44:24\", 0.799, 0.3260745f0)\n",
      "(ep, time, acc, ls) = (15, \"15:44:27\", 0.746, 0.37070704f0)\n",
      "(ep, time, acc, ls) = (16, \"15:44:30\", 0.786, 0.32399616f0)\n",
      "(ep, time, acc, ls) = (17, \"15:44:33\", 0.771, 0.31654957f0)\n",
      "(ep, time, acc, ls) = (18, \"15:44:36\", 0.787, 0.26464257f0)\n",
      "(ep, time, acc, ls) = (19, \"15:44:39\", 0.776, 0.30803627f0)\n",
      "(ep, time, acc, ls) = (20, \"15:44:42\", 0.765, 0.39469373f0)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Chain(Conv((3, 3), 3=>8, relu), BatchNorm(8), #27, Conv((3, 3), 8=>4, relu), BatchNorm(4), #28, flatten, Dense(196, 80, relu), Dropout(0.5), Dense(80, 40, relu), Dropout(0.5), Dense(40, 3), softmax)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainedModel = trainModel(buildModel1())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "testModel (generic function with 1 method)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#This is the testing function\n",
    "testModel(model) = accuracy(x_test,y_test,model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.751"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testModel(trainedModel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task 2**: Attempt to create a different model archiecture - use two additional convolutional layer. Try to maximize the validation accuracy and we'll see in class who gets the best test accuracy (you can only check the test accuracy once)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra: Some Localization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size(digitsMNISTtrain_x) = (28, 28, 60000)\n",
      "size(digitsMNISTtest_x) = (28, 28, 10000)\n",
      "digitsMNISTtrain_y[1] = 5\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\"\n",
       "     width=\"150mm\" height=\"150mm\"\n",
       "     viewBox=\"0 0 28 28\" preserveAspectRatio=\"none\"\n",
       "     shape-rendering=\"crispEdges\" stroke=\"none\">\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"0\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"1\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"2\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"3\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"4\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"5\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"6\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"7\" fill=\"#151515\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"7\" fill=\"#878787\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"7\" fill=\"#FFFFFF\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"7\" fill=\"#BDBDBD\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"7\" fill=\"#6A6A6A\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"7\" fill=\"#0D0D0D\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"7\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"8\" fill=\"#151515\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"8\" fill=\"#A6A6A6\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"8\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"8\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"8\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"8\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"8\" fill=\"#B8B8B8\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"8\" fill=\"#0A0A0A\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"8\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"9\" fill=\"#363636\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"9\" fill=\"#C9C9C9\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"9\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"9\" fill=\"#ECECEC\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"9\" fill=\"#A7A7A7\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"9\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"9\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"9\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"9\" fill=\"#565656\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"9\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"10\" fill=\"#CFCFCF\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"10\" fill=\"#F9F9F9\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"10\" fill=\"#A5A5A5\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"10\" fill=\"#090909\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"10\" fill=\"#010101\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"10\" fill=\"#161616\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"10\" fill=\"#F0F0F0\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"10\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"10\" fill=\"#565656\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"10\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"11\" fill=\"#787878\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"11\" fill=\"#FCFCFC\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"11\" fill=\"#8F8F8F\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"11\" fill=\"#1A1A1A\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"11\" fill=\"#C5C5C5\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"11\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"11\" fill=\"#F5F5F5\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"11\" fill=\"#0D0D0D\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"11\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"12\" fill=\"#B6B6B6\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"12\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"12\" fill=\"#737373\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"12\" fill=\"#575757\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"12\" fill=\"#8D8D8D\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"12\" fill=\"#F0F0F0\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"12\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"12\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"12\" fill=\"#AFAFAF\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"12\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"13\" fill=\"#A2A2A2\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"13\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"13\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"13\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"13\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"13\" fill=\"#E9E9E9\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"13\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"13\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"13\" fill=\"#B8B8B8\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"13\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"14\" fill=\"#0F0F0F\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"14\" fill=\"#9B9B9B\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"14\" fill=\"#BFBFBF\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"14\" fill=\"#8A8A8A\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"14\" fill=\"#797979\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"14\" fill=\"#2E2E2E\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"14\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"14\" fill=\"#FAFAFA\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"14\" fill=\"#414141\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"14\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"15\" fill=\"#8D8D8D\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"15\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"15\" fill=\"#B3B3B3\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"15\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"16\" fill=\"#2E2E2E\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"16\" fill=\"#FBFBFB\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"16\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"16\" fill=\"#181818\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"16\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"17\" fill=\"#363636\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"17\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"17\" fill=\"#F9F9F9\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"17\" fill=\"#121212\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"17\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"18\" fill=\"#ADADAD\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"18\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"18\" fill=\"#B1B1B1\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"18\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"19\" fill=\"#4C4C4C\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"19\" fill=\"#FDFDFD\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"19\" fill=\"#F6F6F6\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"19\" fill=\"#2C2C2C\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"19\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"20\" fill=\"#7A7A7A\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"20\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"20\" fill=\"#C7C7C7\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"20\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"21\" fill=\"#0D0D0D\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"21\" fill=\"#E4E4E4\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"21\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"21\" fill=\"#757575\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"21\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"22\" fill=\"#6B6B6B\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"22\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"22\" fill=\"#D2D2D2\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"22\" fill=\"#050505\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"22\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"23\" fill=\"#999999\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"23\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"23\" fill=\"#CCCCCC\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"23\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"24\" fill=\"#3C3C3C\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"24\" fill=\"#EEEEEE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"24\" fill=\"#C9C9C9\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"24\" fill=\"#282828\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"24\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"25\" fill=\"#A1A1A1\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"25\" fill=\"#FEFEFE\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"25\" fill=\"#9F9F9F\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"25\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"26\" fill=\"#212121\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"26\" fill=\"#F8F8F8\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"26\" fill=\"#A6A6A6\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"26\" fill=\"#131313\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"26\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"0\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"1\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"2\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"3\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"4\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"5\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"6\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"7\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"8\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"9\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"10\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"11\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"12\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"13\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"14\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"15\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"16\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"17\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"18\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"19\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"20\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"21\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"22\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"23\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"24\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"25\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"26\" y=\"27\" fill=\"#000000\" />\n",
       "<rect width=\".81\" height=\".81\" x=\"27\" y=\"27\" fill=\"#000000\" />\n",
       "</svg>"
      ],
      "text/plain": [
       "28×28 reinterpret(Gray{N0f8}, ::Array{N0f8,2}):\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)  …  Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)  …  Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)  …  Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " ⋮                                 ⋱                   \n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)  …  Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)  …  Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)\n",
       " Gray{N0f8}(0.0)  Gray{N0f8}(0.0)     Gray{N0f8}(0.0)  Gray{N0f8}(0.0)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using ImageCore\n",
    "digitsMNISTtrain_x, digitsMNISTtrain_y = MNIST.traindata()\n",
    "digitsMNISTtest_x,  digitsMNISTtest_y  = MNIST.testdata()\n",
    "@show size(digitsMNISTtrain_x)\n",
    "@show size(digitsMNISTtest_x)\n",
    "@show digitsMNISTtrain_y[1]\n",
    "MNIST.convert2image(digitsMNISTtrain_x[:,:,134])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAVxElEQVR4nO3db2yV5fnA8eswAlM0LkPSRUYpasFgJ5quSzMxLHHGbpmbIbDggJG5zBlwZtkrXkwdicOYOZM5IVvMRoyJJl3SxEWjnXNI0kWh0vkvsQ5KWyRYyhiLOJTQ9P69+GXNXA9qywVt9fNJ7hdc56HnJk/Qb57zcJ5KKaUEAABppk30BgAAPm4EFgBAsgkJrOPHj0dXV1ccP358It4eAOCMmj4Rb9rd3R2NjY0T8dYAAGM21lvWfUQIAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkGzMgfXee+/FjTfeGAsXLowrr7wyWlpaoq+vLyIiBgcHo6WlJerr66OhoSE6Ojqy9wsAMOmN6wrWLbfcEm+88Ua89NJL8Y1vfCNuueWWiIjYuHFjNDc3x549e2Lbtm2xevXqGBoaSt0wAMBkN+bA+vSnPx1f//rXo1KpREREc3Nz7Nu3LyIiWltbY8OGDRER0dTUFDU1Na5iAQCfOKd9D9YDDzwQN9xwQxw5ciSGh4djzpw5I6/V1dXF/v37T/ctAACmlOmn85s3b94ce/bsid/85jfx7rvvjlzV+o9SymltDgBgKhr3Faz77rsv2tra4qmnnopzzz03Zs+eHRERhw8fHjmmv78/amtrT3+XAABTyLgC6/7774/HHnssnnnmmfjMZz4zMl+5cmVs2bIlIiI6OztjYGAgli5dmrJRAICpolLG+DnegQMHYt68eXHxxRfH+eefHxERM2fOjJ07d8ahQ4di7dq10dvbGzNmzIitW7fGsmXLRv2Mrq6uaGxszPkTAACcYWO97WnMgZVBYAEAU8lYc8k3uQMAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyU4rsDZt2hSVSiVee+21iIgYHByMlpaWqK+vj4aGhujo6EjZJADAVDLuwOrq6ooXXnghamtrR2YbN26M5ubm2LNnT2zbti1Wr14dQ0NDKRsFAJgqxhVYJ06ciA0bNsTWrVujUqmMzFtbW2PDhg0REdHU1BQ1NTWuYgEAnzjjCqw777wz1qxZEwsWLBiZHTlyJIaHh2POnDkjs7q6uti/f//p7xIAYAoZc2A9//zz0dnZGevXrx/12n9fzYqIKKWMf2cAAFPUmANrx44d0d3dHQsWLIi6uro4cOBAXH/99bFr166IiDh8+PDIsf39/e+7RwsA4BOhnKb58+eXV199tZRSyrp168pdd91VSill165dZd68eeXkyZOjfs/u3btLRFiWZVmWZU2JNVbTI9G9994ba9eujfr6+pgxY0Y88sgjMX166lsAAEx6lVLO/o1SXV1d0djYeLbfFgBgXMaaS77JHQAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBINq7AOnHiRNx2221RX18fl19+eaxZsyYiIgYHB6OlpSXq6+ujoaEhOjo6UjcLADAVTB/Pb9q4cWNMmzYt/v73v0elUom33nprZN7c3BxPP/10dHZ2xooVK6KnpyemTx/X2wAATE1ljN55551ywQUXlGPHjo16bdasWWVwcHDk101NTWX79u2jjtu9e3eJCMuyLMuyrCmxxmrMHxH29PTE7Nmz4+67744vfvGLcc0118Szzz4bR44cieHh4ZgzZ87IsXV1dbF///6xvgUAwJQ25sA6efJk7Nu3LxYvXhwvvvhiPPjgg7Fq1aoYGhqKSqXyvmNLKWkbBQCYKsYcWPPnz49p06bF6tWrIyJiyZIlsWDBgnj99dcjIuLw4cMjx/b390dtbW3SVgEApoYxB9aFF14Y1157bbS3t0fE/0dUb29vLFq0KFauXBlbtmyJiIjOzs4YGBiIpUuX5u4YAGCyG/NdW6WUnp6esmzZstLQ0FCWLFlS2traSimlDAwMlOuuu65ceumlZfHixeW5556r+vvd5G5ZlmVZ1lRaY1UpE3CjVFdXVzQ2Np7ttwUAGJex5pJvcgcASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBkAgsAIJnAAgBIJrAAAJIJLACAZAILACCZwAIASCawAACSCSwAgGQCCwAgmcACAEgmsAAAkgksAIBk0yd6AwCQ5aGHHqo67+/vHzV78sknz/R2xu2tt96qOh8YGDjLO2G8XMECAEgmsAAAkgksAIBklVJKOdtv2tXVFY2NjWf7bQH4mPvrX/9add7c3Dxqdqr//VUqlarzasefqWP7+vqqHnvo0KGq8+9///ujZt3d3VWPZXzGmkuuYAHAh5iAaxFMcQILACCZwAIASOYeLAA+9tatWzdqtmjRoqrHLlu27Exv50Odf/75VeeXX3551fnOnTtHzb785S+n7umTzj1YAAATTGABACQTWAAAyTyLEICPvYcffniitzAmV111VdV5Z2dn1flrr712JrfDOLiCBQCQTGABACQTWAAAydyDBQCTTFdXV9X58PBw1fkf//jHM7kdxsEVLACAZAILACCZjwgBYAL95Cc/GTU71UeBd9xxR9X5E088kbonTp8rWAAAyQQWAEAygQUAkMw9WABwFvz0pz+tOt+0adOo2QsvvFD12N/97nepe+LMcQULACCZwAIASCawAACSuQcLAJJV+26ravdaRUSUUkbNli9fXvXYQ4cOnd7GOGtcwQIASCawAACSCSwAgGTuwQKAcbr11lurzn/xi1+MmvX19VU9ds2aNaNm7rWa+lzBAgBIJrAAAJL5iBAAxumyyy6rOq/21QvVZhER9fX1H2kWEbFnz56q86NHj46adXd3Vz2Ws8MVLACAZAILACCZwAIASOYeLAAYp9bW1qrz48ePj5otW7as6rEPPvjgqNl5551X9dhT3cdVqVQ+8rHf+ta3Rs2eeOKJqscyfq5gAQAkE1gAAMkEFgBAsko51Ye0Z1BXV1c0Njae7bcFgEln0aJFo2Zz586teuypvh/rmmuuGTVbtWpV1WN37tw5anb11Vd/0BaJU9/TdiquYAEAJBNYAADJBBYAQDLfgwUAE+iNN974SLOIiL/85S9V53v37h01u+mmm05vY5wWV7AAAJKNK7Da29ujsbExrrrqqmhoaIiHH344IiIGBwejpaUl6uvro6GhITo6OlI3CwAwFYz5I8JSSnznO9+J7du3xxVXXBF9fX1x2WWXxfLly2Pjxo3R3NwcTz/9dHR2dsaKFSuip6cnpk/3SSQAnCk33HDDqNmpvlZg8+bNZ3o7xGncg/Wvf/0rIiLefvvtmD17dsycOTNaW1ujt7c3IiKampqipqYmOjo64itf+UrGXgEApoQxB1alUonW1tZYvnx5zJo1K44ePRptbW1x7NixGB4ejjlz5owcW1dXF/v370/dMADAZDfme7CGhobinnvuiccffzz6+/vj2WefjXXr1kXE6Kd5T8CXxAMATLgxX8F66aWX4uDBgyNfq9/U1BQXXXRRvPLKKxERcfjw4ZGrWP39/VFbW5u4XQD45Prud79bdX777bePmv3n/8v/68knn0zdE9WN+QrWvHnz4sCBAyPf0bF3797o6emJhQsXxsqVK2PLli0REdHZ2RkDAwOxdOnS3B0DAExyY76CVVNTE7/97W9jxYoVMW3atCilxNatW2Pu3Llx7733xtq1a6O+vj5mzJgRjzzyiH9BCAB84oyrfm666aaq3xBbU1MTf/rTn057UwAAU5nLSwAwyXzuc5+rOv/hD39YdV7tH5W1tram7omx8agcAIBkAgsAIJnAAgBI5h4sAJhkfvnLX1adNzc3V53//ve/HzXzzMGJ5QoWAEAygQUAkExgAQAkcw8WAEyghx56aNRs1apVVY999NFHq85/8IMfpO6J0+cKFgBAMoEFAJDMR4QAME633npr1fnSpUtHzbq7u6see/PNN4+a3XvvvVWP/dWvfjWG3TGRXMECAEgmsAAAkgksAIBk7sECgHHauXNn1fkdd9wxanb48OGqx371q18dNdu+ffvpbYwJ5woWAEAygQUAkExgAQAkcw8WAIzT3/72t6rzuXPnnuWdMNm4ggUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACQTWAAAyQQWAEAygQUAkExgAQAkE1gAAMkEFgBAMoEFAJBMYAEAJBNYAADJBBYAQDKBBQCQTGABACSrGli333571NXVRaVSiddee21kPjg4GC0tLVFfXx8NDQ3R0dEx8trx48fjpptuiksvvTQWLlwYbW1tZ373AACTUNXAWrFiRXR0dMT8+fPfN9+4cWM0NzfHnj17Ytu2bbF69eoYGhqKiIj77rsvZs6cGXv37o329vZYv359HD169Mz/CQAAJpvyAebPn19effXVkV/PmjWrDA4Ojvy6qampbN++vZRSyuLFi8uuXbtGXlu5cmXZtm1b1Z+7e/fuEhGWZVmWZVlTYo3VR74H68iRIzE8PBxz5swZmdXV1cX+/fsjImL//v3vu+L1368BAHySjOkm90ql8r5fl1JO+fr/vgYA8EnxkQNr9uzZERFx+PDhkVl/f3/U1tZGRERtbW309fVVfe1/vfvuu+PZKwDAhOjq6orjx49/5OPHdAVr5cqVsWXLloiI6OzsjIGBgVi6dOmo13p7e2PHjh3xzW9+s+rP+e8QAwCY7BobG6O7u/sjH18pVT7L27BhQzz++OMxMDAQF154YZx33nmxd+/eOHToUKxduzZ6e3tjxowZsXXr1li2bFlERPz73/+Om2++OXbv3h3Tpk2LzZs3x4oVK6q+6T/+8Y9ob2+Purq6OOecc8b5RwUAOHsuu+yyOPfccz/SsVUDCwCA8fNN7gAAyQQWAEAygcUo7733Xtx4442xcOHCuPLKK6OlpWXkHyZ80OOSmFw2bdr0vsddOXdTw4kTJ+K2226L+vr6uPzyy2PNmjUR4fxNFe3t7dHY2BhXXXVVNDQ0xMMPPxwRzt9kdMYfCzjmryblY+/dd98tTz75ZBkeHi6llPLrX/+6XHfddaWUUr73ve+Vu+66q5RSyq5du0ptbW05efLkRG2VU9i9e3dpaWkptbW1I09jcO6mhh//+MflRz/60cjfv4MHD5ZSnL+pYHh4uHz2s58tL7/8cimllN7e3jJz5szy9ttvO3+T0I4dO8qbb7456qk1H3SuNm3aVNatW1dKKWXfvn2lpqam/POf/6z68wUWH6qzs7NccsklpZQPflwSk8N7771Xmpuby759+973Hw7nbvJ75513ygUXXFCOHTs26jXnb/L7T2Dt2LGjlFLKyy+/XC666KJy4sQJ528SO1OPBZx+Ji+/8fHwwAMPxA033PChj0ticrjzzjtjzZo1sWDBgpGZczc19PT0xOzZs+Puu++OP//5z3HOOefEz372s7jyyiudvymgUqlEa2trLF++PGbNmhVHjx6Ntra2OHbsmPM3RWQ+FtA9WHygzZs3x549e+LnP/95RHz445KYWM8//3x0dnbG+vXrR73m3E1+J0+ejH379sXixYvjxRdfjAcffDBWrVoVQ0NDzt8UMDQ0FPfcc088/vjj0d/fH88++2ysW7cuIvz9m0qyHgsosDil++67L9ra2uKpp56Kc88990Mfl8TE27FjR3R3d8eCBQuirq4uDhw4ENdff33s2rUrIpy7yW7+/Pkxbdq0WL16dURELFmyJBYsWBCvv/56RDh/k91LL70UBw8ejKuvvjoiIpqamuKiiy6KV155JSKcv6kg87GAAouq7r///njsscfimWeeic985jMj8w96XBITb+PGjXHw4MHo6+uLvr6++PznPx/t7e3xta99zbmbAi688MK49tpro729PSL+/z/evb29sWjRIudvCpg3b14cOHAg3njjjYiI2Lt3b/T09MTChQudvykk67GAbnJnlDfffLNERLn44ovLkiVLypIlS8qXvvSlUkopAwMD5brrriuXXnppWbx4cXnuuecmeLd8kP++edO5mxp6enrKsmXLSkNDQ1myZElpa2srpTh/U8Wjjz5aGhoayhVXXFG+8IUvlMcee6yU4vxNRuvXry9z584tn/rUp0pNTc3IP+b6oHP1zjvvlG9/+9vlkksuKfX19eUPf/jDKX++R+UAACTzESEAQDKBBQCQTGABACQTWAAAyQQWAECy/wOHbfem9TF5cwAAAABJRU5ErkJggg=="
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots; pyplot()\n",
    "totalDim = 100\n",
    "img = zeros(100,100);\n",
    "x , y = rand(1:72),rand(1:72)\n",
    "img[x:(x+27),y:(y+27)] = digitsMNISTtrain_x[:,:,45]\n",
    "heatmap(img',yflip=true,legend=false,color=:greys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Random\n",
    "Random.seed!(0)\n",
    "\n",
    "dataLen = 1000\n",
    "xData = zeros(totalDim,totalDim,1,dataLen);\n",
    "yData = zeros(13,dataLen);  #pc, x, y, 10 classes  (we assume 28x28)\n",
    "\n",
    "for i in 1:dataLen\n",
    "    x , y = rand(1:72),rand(1:72)\n",
    "    if rand() < 0.8  #w.p. 80% have some image\n",
    "        xData[x:(x+27),y:(y+27),1,i] = digitsMNISTtrain_x[:,:,i]\n",
    "        yData[1,i] = 1  #pc in label \"yes image\"\n",
    "        yData[2,i] = x\n",
    "        yData[3,i] = y\n",
    "        yData[4+digitsMNISTtrain_y[i],i] = 1\n",
    "    end\n",
    "    xData[:,:,1,i] += 0.2*rand(totalDim,totalDim) #some noise\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB43klEQVR4nO29bczfZXn/f7SFci+MQi/o/d3VFqygqV3IZIoxzmrmYhwYDTA292AJupvEByNZopg5DYk/HzgxzixxC5tGtpBgYraiTlm6Oak23jDp7XXTltKCpcg9FPr9PyD0z5fv69X1hM/F97ro+5XwwLOnn+95c5zn53Odx/s8jlm9Xq9XIYQQQgihM2YPuwEhhBBCCK838oEVQgghhNAxQ/nAevLJJ2vr1q315JNPDuPnQwghhBCmlFOG8aPbtm2r9evX19y5c2v27P//G+/CCy/E+kePHh0oO3jwINZdtWoVlj/xxBMDZU8//TTWpQ+/Z555Buu+4Q1vGCg766yzsO7DDz+M5fTsc845B+s+8sgjA2UXXHAB1j3lFJ7eOXPmDJTdf//9WHf58uUDZTYWR44cGSg788wzse7evXuxnOZ6yZIlWJd49tlnsdzm5LnnnhsoO+2007DuxMTEQNm8efOwLs2T1T3jjDOw/MEHHzyh51ZVnXfeeQNlJq8km62qmjt37kAZ9bmq6uKLLx4oe/zxx7HuqaeeOlBmdmHzRzz11FMn/Hu0/quq5s+fj+X79u0bKDv99NOx7rnnnjtQdvjw4RP+PVtPjz32GJbTONveSfZN67SqavHixVhO++RL9+2XQm02OyQboLmrqhofH8dyskOzrUOHDg2Utazfs88+G+vSflrFY2H7+p49ewbKZs2ahXXNDi+66KKBMnvHES39sHVqY0820PJOpb2pytcZjbPtT/TOsXfqfffdh+VGXIQhhBBCCB2TD6wQQgghhI4ZiovwRc4999y+oz86zq7io0RyXVVVjY2N6W+9HHOVEL/xG7+B5XYkSseOCxcuxLoPPfTQCbeN3JfmYqI2VLGr0o5E6VjVxoKOo80VuHLlSiynI1/rHx2hHzhwAOuai4javGLFCqxLrkobY4JcflXs3qviY27TLVL5+eeff8Jtq2KX8vPPP491ye6tbY8++uhAma11c4tQ28y9Qy4wcyVYO0hqYC4Gahu5a6rY3WJjPDIyguXUZnsGjYW5W8hNVcVzYjZLLj5bIyRLsH7YmqT6+/fvx7rk+ifbrOI2P/roo7hfmA0RJtsge6H3QpXbIbXN1hPthzan9FyTUdh7i9yz5pIk179JFUZHR7H817/+9UCZ9Y/2LdurW8kJ1hTQ8tINIYQw/WnRM4VQlQ+sEEIIIYTOyQdWCCGEEELHDFWDNXfu3D5frvmnyRdtmiG7akzYkS/5kXfv3o116eouXR2ucj8yXbH+1a9+hXVJB2RX9+3qNulATJdBvna7dk3agNWrV2PdHTt2nPDvmZaItGQW0sGuppMWwXRcFOrBNCN0ldquoJuGg55ha4TGza7S2++RHmXp0qVYlzRNpssg/aNhV7epbaTrqOL1u2jRIqxrV95pjEwTQ2Nk19hpTlqutldxm239ku7ENEoWQoDm1caN5sk0Q/R71gbTStHYmWaT5tT6QdpR0/3aeqK5tlA9tLfYurExomfbfkHrrMVmDZPHtIw9abBMv9yyl9l7kvR9tpe1khOsEEIIIYSOyQdWCCGEEELH5AMrhBBCCKFjZvXM0T+FbN26tdavX1+nnHJKn3+3JT2MxWIyXQb5z60u6bvMf2v+fsK0PaQZMR8wxeew9EDmDyf9TIs+yDQjNEbWD4vPQs+wdCZkui0agCrWUNlc0zzZ/JOGzvSBFg+G7MXqkp7BNBXW5gceeGCgzMae9B7Lli3Dui3pb1rs3uaa7MI0eKZzIRu3Z1AcHdMMkZ7QdHyWmoU0TRZ/yDRthLWDtGc2bmQXlqaKtKo2xi1aR7LjKt7XbR+ydUbs2rULy6l/LWvddHX2niR7sXcOvT8pPVQVt9l0xrZ+yQ5t/dL+ZGmx7F1Ebbb3Pe3V9lyLp2jkBCuEEEIIoWPygRVCCCGE0DH5wAohhBBC6JihxsG6+OKL+3zok5OTWI80RhZTyvzTlKPQNDEHDx484ecSpnGx2F3ki7b4JaRzMY2LaYmIllguFheF9BOWL8787/RsyitVxXNiGgfrH2lGLHYXaS1MX0JjYToCsrcqts8W3YLFGrvkkkuwnMbIbJlix1isOIpNZnVNS0TaJYtL1aKhNI0S9dt0ihRPz9YIaV9MBtuSV9X6R3NqOQ7NXihvqOmDaNxsLyM9i2mtLJck7X2mq6KYYDantM4sV+PatWuxnDQ/puEhjbDtWfYM0hJZDMGW/Jlky5arryVnpO3VNH+mibI9oCXGGtlAS57i45ETrBBCCCGEjskHVgghhBBCxww1TMN5553Xdzxnx+p07GxuMbsmev/99w+U2fEpXXel9DJVfuxIR6h2tZWudNsVbXI92dVmaxtdY7ZnkFvL3H4taSvsGjuNm/0ezbUdO5sriMbZ3BHk3rHUF+TWtGNus2Vqm9ksXbG2Y26zLUoPYr9H/TMXKIV6sPQp5t4hN6PZEF1Nt360uDrMNUP9Ntcx2YBdH7d5ojVFc1fFKYKsz7Z2qH9Wl9zaLWmq7LnmyqPXl6UZI7emrXVLi0N2a3sAhZCwPYBCfVhdC5FBY2RjQXuqveOobWbfNhbkIrRPD3pPtoRkqWIXqPWPbM5kKZZez8gJ1hRgG1gIIYSZif1REIKRD6wQQgghhI7JB1YIIYQQQscMNUzDeeed1+enthQH5O81vYf56hcsWDBQZvog8jmbNoT0M+eddx7qyUzDQVoZ82WTv9jGwq5H0zV2uzZPWgS7Sk3XmE37YvoCapv5zukZVtd0IKRzMRvav3//QJnZEIXTMP+9aaVIY7Rz506sS1fQrW2mW6CxsKviLSEdaE4WLlyIdc0NQ+vMrmjTfmHr6cILLzzh3zO7IL0GzUcVt5m0OlWuSaV90q7YU5stxILpV2ntmCaGsBASpCVqCd9SxXu16bis34TtcYSFJqAwG/YeoX3SdE6WmofabCFnaJ1ZuBiyC9MT2zuHNIK2Rmhvsb3TdMZU37RrFB7KUoS1khOsKcA2gxBCCCGcHORLIIQQQgihY/KBFUIIIYTQMUPVYD3yyCN9GgHzOZNP3WLgmA+4JdYF+fVNM0LPNd2KQbFKTBtCMXMoPlOVpx4irYz1j/QzFpeKsLa1xN0yjZJpRgiba4qPZnZIc2J6Dxo301+YRoX0BTZPNG7UtyrvX4v+jTQjtG7sGfbcFg2W6Y5ob6A4WlWutaFnmOuf1rvZJpVbXbMX0oeYdonmxHSRLel2TPtC49aiX7X5sPVL68G0PaRds32WxtPWjdkszZPp7QgbC7ND2pctxh6V23NbUnaZTpFszmIT0jjbnBqkM21ps72fWskJVgghhBBCx+QDK4QQQgihY/KBFUIIIYTQMUPVYJ166ql9sUks1gX5301/0ZLXy7QI5Je3tpGf3OKt2DNIg7F9+3asSzoZis9U5foC8jmb/5381uY7Jz+5aSds7Mn/bpo2Gnura5ofivNlqY4ojllLTi7T2pi90DjbeJImxrQvppUiDY5pH8jmLLYZrVWLxWXzRGNk/SD7bs1RSTpF0yjRmrS2kU7GYnS15PBbunQp1qW4W5bj0PYAsnEbT7JP01DSGFsbTI9EOkN7N9D6M/0UaXlb9JZVvCatf2Rbtn5JE1XFe5/FlqQxsr2M4mORzVdVjY+PYzmNvf0excwyzZ/FliTtr62nlrXeSk6wQgghhBA6Jh9YIYQQQggdM1QX4SmnnNJ3nG9HonQ8ODExgXVXrlx5wr9/+PBhLKfjTHNpHDhwAMvJPddyddtSHNBxtB07t1xtteu8dORvbhW6gtzr9fAZdsTcEk6DXFJ2tGuhJejY2FwaZC92zE3H9ZbSyOaP6ltdss8Wt1EVu0XMhULPNlcXjRuFJqny9Us2Z6FaaP7MPWDl5g4iyIVia49cWvZbFuKE5slsizB3m6VKob7YHrBo0aITbgeta7vmb5DUwFz8tEZs3Kyc9i2TYtDv2XuLUmuZ69HCDZA8xvZZkquYvdHvPfjgg2hHIyMj+AzCbMjaTNjv0ZqytU7vPksF1EpOsKYAW3AnI/ZxFEIIM4mWF//rnZZcjScz+cAKIYQQQuiYfGCFEEIIIXTMUDVYzz//fJ9WxTRK5E+1q6qmyyC9Dl3PrGLft2kDSNtz9OhR1I21XEG3MA3kfqRQA1VV9913H5aTbsFSOJCmwq6g0/Vv066Z3o7mz9If7du3b6DMtGumOyJNhGkRSLdgITIWLFgwUGb9MLtvCfVA2qwdO3Zg3UsuuQTLSRNhoUWobaZxIDexrT37PbJPC/VBY2ThO1q0UqbLoKvi5kKh9CmmG7Tyc889d6DMNEO0d1ooG7N72+MIsqGW0ASmlbRr+qSBtb2FQhNQSIAqnj9be5aSit5RNHdV3GaTV9he1hJeiGzAbJbs3t5lLSF8TL9KdmF7i7WDbM7eRbSXmX61lZxgTQG2GYQQQgjh5CAfWCGEEEIIHZMPrBBCCCGEjpnVM8fkFLJ169Zav359XXTRRX36oxYNlvmnTUtE5Xv27MG6dB3XQvKTH9liSpk2gOqbZoQwbY+NBf2e6S9Ix2OpKEgHYhoA6x/1xVJ7kCvWfs9iDVG5zRP59S2eE7XD/Pqmn6H+mW1RO0yPaGFEaOwtthVhukhi586dWG76MIqlZTG6WnROLbHpzGZbUk9Rm82+x8bGsJx0kaYFJJ2ahRvYvXs3lhO2t1C/Ta9FdmipXWyeyC5WrFiBdWmdmSaK9J2tMdNIq2o6PnqGabta9rIWnaKt9Zb4ePYeoc8Ms3vbwwnTjdFcm3SH2maaa4tjZuQEK4QQQgihY/KBFUIIIYTQMfnACiGEEELomKHGwXr88cf7/MamGaE8TebXNx8+6YYsDgf5nFv87KbVMH8/5U6zHIeEyehIR1BVtWTJkoEyiwezatWqgTLL4Uh+a9O+WM450p1YzCTSDJg2wHzqFEPMtHnUDoqvVsU6gl27dmFdg+zF+kc2ZzZr+iBaDzb2pH2wtpF+0XRgpmlr0UlQXDkbixYNjmmXSI9k8YBMy0mYfob6Z+NJuiHTa9kzKGaS7Z1kF6YbpDEyjVJLHDPbc6htlneU5oneQ8drG2ExyEgfZHomW5NkFzbXtB4WLlyIdSnulsUrsxhdtOfYM+j9a7pYe2/RM0w32LJ3tpITrBBCCCGEjskHVgghhBBCxww1TENV/9GfHcHbMS65YewIno5V7QjWjrTp2NGOKKnNlmKGjsrtCN6gttkzqB12NdZCCNDxrrlnKTyBHePTc81dakfM1maCXAwtx9zmprIUDmRzFnJkZGQEywmbP3q2pYeh+bM2kHvWjuBtnshlY+uJXCh2bd72EbILu+ZN7rKWrdL2EHOv0xhNTk5iXXIFmavTXB00VxamgfYRSgVVxWFdrA0UTqOK3UE212TLFm7AXOMt4RuobTQfVSx5MUmB7VkUTsHGk1z05r60ctqfWtxzJv2gZ9jas9+jsBUt7zgLI0PvX9u/W8LWVM3gE6yWWBldYLGmCPsgnCpa2tYF9sIkusrpdKK0fFy91tgH/VRhH27TAdtcp4rpbBct66kL7EN4OmAv16mixS5e67ZZ7K+pomV/sj+EpgqLCTbdmbEfWCGEEEII05V8YIUQQgghdMxQwzS84Q1v6HNvteiO7Do3pZGoYm2WhYWgEBDmZ7e0MeTjNr83XU22I1iqa3oP6x9pSex4uOUKK+kyTEdi6SxI02S+c9IcmAbPriCTtsNcKKSfMF0OpYEwt7bpnEgTMzo6inWpzXasbvoZWn92rZxswK7Y0zzZWNhc03o3lw1pycwlaaEQyI5a3EmmAyNtl+myzEXUEkaGxtnCt9hYEKahJEmAzRPtT7aXWZoTcoOb1orGyHRgBulBbdxoD2gJs9ISCsPKbX9q0RPSHmD2ZholGguzQ7Ihe6+bbdFYmI6P5qRlLRyPnGBNAS1xUUIIIUx/WmIThlCVD6wQQgghhM7JB1YIIYQQQscMVYN19tln9+kMJiYmsB75os0NZ9oO0gxYagjy95tv2XQSpNewNpOP20IvULmlQ7B0QqS3Mt0RaaJa/PqmRbF5onILN0Bj35pqg2zOdE7Uv/HxcaxLaTVME0Wpi6pYX2DaQ5prm1OL80VzZTZEcWasbWRvprWxcppX+z2KjWT6KdMeUhwc03vQs027Rm02m7U1QvoX0x3RnLbGbiM9mmliaP5MS2S6VuLBBx/EctLamK6K1q+5/ei5tpfZO4DeOabBIs2PvXNsPKnfFtuM+mfPJfu0mI7WZrIts1nS7Vp6IJtr0vfZ/NF6aElpdTxygjUFTOeYOyGEEEKYevKBFUIIIYTQMfnACiGEEELomKFqsObOndvnv7aUERR/yGKBmG+4JXQC6QgojkcV6wgsRohpRsi/bDE7SNtBuZSqqpYvX47lpnUjSI9mMbpIu2SaGvO/03hazB3S2phuxfRIFI/LbIviFZlea8+ePQNlNh+mL1i1atVAWYvGwbQ95sKmNptGidak2SyNm2m7LAccaVesLukvLNaU2QXpUUxvRxoOi9tEabRMb2l7FrXZNEpkW7aXmUaF9jOzQ3qG2QVp2qwNtkZoLEz/Rs+weEdkQ6Zds3hOpFMzuyd9mO1ltt/T3mm5Acm2TGNIbW7RcVbxe8D0aLR2bJ5sXVMsrbGxsRP+PXuHt5ITrBBCCCGEjskHVgghhBBCx+QDK4QQQgihY4aqwXriiSf6/PPmqyffucX3ME0MaSos7gf5300/RXndzD9tMTvIp96SC8k0HBY/ijRG9gzSSrXE/jI9hOmAaDxNM3Lw4MGBMtNJ2FiQzsF0RzRGpjEjfZDFKyM9k/2exW4jPYPp3Ey71BJLi9akxY6x8SRMQ0lttjVJc2KxxiwOEtkh6aeqeK2anoXKrQ2mSaX1Z/ksaSzM3kzzRflWTadINmdrhPphWjLbL0jXaHsZYWuEcpdanDCLCUZtbtGSWS5Ze2/R+jXdEc2pxekj27J3ma1J0mbZWqcxsjVi7aBvCXun0rvB9tlWcoIVQgghhNAx+cAKIYQQQuiYoboIH3roob4jc7tWTi63xYsXY9377rsPy+m6qrmNqNyu2JN7b/78+eiqsiNmcj2Yu5Su4tp13pa0I3btmo7brW10zL1w4UJMfWDH3HTEbC4NOvK1Y25KXVPFLhtzR9BY2JzScb25jcz1RJjLlcbCro/b2JPLxvpHdc2dSONmIQ+sbfRsc0mSS8OuXZMrqIrdM+ZieOCBBwbKzHVBbW5NJ0VX+m2tk82Ra73K918LC0DQvJqLn67HWxgDG3vqn8kEaOzNDWv7IbmvaH+rYre2he8g9yyFGjje79HasXREtDeYK5/WTktIhyqeE3JTGlbX3kX0eybdIfs0m20lJ1hTgC3wkxHbDEIIYSZh+qkQjHxghRBCCCF0TPMH1tNPP10f+MAHavXq1fXmN7+5Nm7ceCwy+IMPPlgbN26s0dHRWrduXW3evLnr9oYQQgghTHtm9cihfxyefvrp+o//+I9673vfW7NmzaovfelL9a1vfavuuuuu+uhHP1pLliypm2++ubZs2VJXX3117d69e0CDsXXr1lq/fn1dcMEFfe408wGTvsB8vaYZIX+4XUsmzYHpclp0Y6ZRoavJdk2U2mH+cNOdUIoZg1IRmN6D/OE2p9Zmmj9rL13zNb2IzTU92zQqpJWyq8Y0f6brsLAXZLOWGoL6ZzZkbaY1ZW0j+zbdEWmJrK6ta7qmbXbYos2zdpDWzeyQfs/mieranmUuKbJPmyfS/FgYA2tzi86JND8tKX9sjZiWiPrXoou0UAikwbHULrbHUSgL0/1S28w2zS5IY2b9I91fS8omm1PTk+7cuXOgrCXEgtmbjQWNp9kQvTPsHWDfAUbzCdbpp59e73vf+44tuiuuuOJYrI3bb7+9Pvaxj1VV1YYNG2pkZOSkPMUyMV0IIYQQTg5etQbri1/8Yr3//e+vQ4cO1dGjR/v+al62bBkmkA0hhBBCeD3zqsI0fPazn62dO3fWV77ylXrqqacGjpIbvY8hhBBCCK8LXvEH1uc///m644476rvf/W6deeaZx2IVPfTQQ8dOsSYnJzVFRdULftyXfpSZBoD8nuZPNS0C+fZNi0Ixr8znTG2z9AsGxTshn3UV66rMl22QdsXGs0V/QZofi1Vk+jCKS2O6HNIX2Ee9xTGjeDCmIyBtgGnJSBtgsVxMN0bxlVpSONg8WeyfFte2aQ8JGjdbe7YHkHbFYtVQ3DRbk4888giW0xhZGhdqh+1DNNe2FizECa1306JQqhTTkpld0JzYfkg2a6lkqK6lcTJdFcXNW7VqFdbdsWPHQJmlVqN5snEznRNpmmwP2Lt370CZaQwtlhZp2mxN0+8ZZMvWhl27dmE5jZ2tSYrnZfow00TR71l6PYqR2RKb8Hi8IhfhF77whfrGN75R3/nOd/o6cs0119Stt95aVVVbtmypAwcO1JVXXtlJQ0MIIYQQZgrNJ1j79u2rT3ziE7VixYp65zvfWVUvfOH+6Ec/qltuuaWuv/76Gh0drblz59Ztt93WlHgzhBBCCOH1QPPXz6JFi9QNMzIyUnfddderblQIIYQQwkxmqMdLp5xySt8JF/nkq6rWrFkzUPZicNOXYzmrSPNh8UtadAuk7TIdgelLyMfdonExX73l8KN2UP62Kh4jiu9SxToZ04ZYOcVMMv0F6aoszoxpRiyXFUFaGYttRmPUkouyirUrpiOgNWLaLtNJkB6F1oLVtVhTLXoGy8tHWj6zIRp7s+8WO7Q8ibS3mKaxJZ+laaLIDk2LQnZhNm9xnmg8TXtIerQWnZvpmaxtpF813RG9G2wvo3bYWre5bskbS/Zp69d0rbT+Vq9ejXXJ5iz2E7W5i7RBpq2lGHS29swuqL7pVynSQVfp7pIqJ4QQQgihY/KBFUIIIYTQMUN1ET7//PN9VzUtpAOlyTD3l7nn6FqpHUfbtekTrfv444/jcbIJ/ukI1o6j9+/fP1BmR5/WDzp6tqNWcq21PPeZZ57BcTaXDV3dNbcvuYntudZmOkq2umRzVpfcRnPmzEHXSou7m1w+Ve72IyysB7kezL1H/TOXFj332WefbTqGJ1eQuTRoTs2NQ9f8q9hVaWuSNKnWN7oeb+vXnkH2Yi5wWtet4TtoXZu8gtw+tkYoFI21zVxd1Daba3KL2TuAZBu9Xg/H2aQfixcvHihr2ddtnVqbaU4sHEqLZMZCMtBYmOuQ9hFzl7a4Z62c1rW5GRcsWDBQZq7jVnKCNQXYpJ+M2GZwMmK6lZORrjQOIbxW2EfsyUjG4sTIB1YIIYQQQsfkAyuEEEIIoWOGqsHq9Xp9PnsLN0AaB/N729VW8odbqg1y5djv0ZXnRx99FPVW5telZ5iWaOHChQNlFpesRfNlV2Yp7IGNMYUVsD5bmgTSa9i1a5pTcz2Rjq+KtQiW2oOwK/YU1sPG2MaI7NC0cmQvpkccHx/HctIC2XhSm61t1A5b6zYWFIbA1gi12VzVpL+o4qviFJKlivVhpqsj14pphsxeKN2KhUKgsTfti80JrVXSDFWx1sb6QRoj28tMS0S6GgvVQro6q2t7HPXPQvjQ+rUQGWT35oazfZ1+z9YThQGyfYHm31JBWTqplpAVpOMyHZ9pKMle7H1P6dJM491KTrCmgESvDyGE1xdd5acLJw/5wAohhBBC6Jh8YIUQQgghdMxQfVlPP/10n2/VNCMUD8hidphvmPQTpn1oidtEPnV7rsUIaUlbQD510mRUuWbEYrwQdCxu/aO6puuwtEjLli0bKLOwFzRPpu2ydrTorWjsbU7JLkyvZfoC0gGtWLEC67boFE1fQJoPakNV1apVq07o/1/F42bpPmz90nq3NUKaGOuz6YOovj2D0gnR+FTxGNm4mb2QLZssgTQxNsa2B7SkHSF7MQ0ljZFpu0hvWcX9M53T5OTkQJnFXqR9hGKYVfkeQs8wTRStaxuLFl2kvVOpL6ZHpHGz+acUWvYMi/NFelmLxdWio7b3CI2RjVsrOcEKIYQQQuiYfGCFEEIIIXRMPrBCCCGEEDpmqBqsiy66qM8vapoKinVh/lRLR0IaKvs90hKZxoF0C6Z9Mp8ztdmuBNPvtcbRoRgv5teneCKmIyDNj+V/aslbZ/2gOCqmzTMNFo2dxcYhvcfy5cuxLs2TxX0xHQhpc0xf0lLXxojiwZAmroo1HGZDpGewuE2mqSCbpdhYVWxzpiWz36P9xeIE0Xja79HeQPGZqlwHQvZp+jCqSzGQqnzvbMl/SjZg+lXSmJmGsiXWn+kiSatqOjdaq/YOsGeQ1s1slnSDtnfaOqP9ydpG8RQt1hhps0wfZvo+2uMoF2UVvydt7dn+S/Nnuk96B5h+tZWcYIUQQgghdEw+sEIIIYQQOiYfWCGEEEIIHTOrZ87tKWTr1q21fv36Ouuss/piAFn+J4qDZf5i85OTHsW0CNu3bx8oW716NdalGC/WD/Opk2agJeaO6SFMu0QxqMx3Tm0zkyFftmkqTGNGY2eaNvKz2++ZnoXsxTRtpD2zGFZkF6ZdszhI1Gaba9LPkKaqyjVYFFesRQtoeibqX0vewioeT1vr9Gxrm2mlqN+m7aF5td8j/YxpUVr0Jba3kJbE9ifT7NGasj2AxsieS+0w27TxtHh6BOmcbL+gObX91PZOapvZkOmqCIsJRevB9gDSglFOzSrWW5q9mW3RvNo7nPSyFv/PdNS0rs22TBf3autW5QQrhBBCCKFz8oEVQgghhNAxQw3TcMYZZ/QdjVqofnKLmQvNXAx0VG6uGXK3mCvBXDZ0jG9HooQdwdMxdcvV9qqqtWvXDpSZy4awI3E6rrUjcQtZQMfcdnxOKSosxILNH9mL/R7Nif0e2Za5tMwVRK4Vc5eSW9PWk40FucztejRhKVHIBmz+zY3a4n6ma94WgsD2EXLvmGuG1iSFlqlie7FQGDYWZEeWxoXm1Pqxe/duLCdsPCkUgrnxyM1ke4uFWaH9d8+ePViX3L4WLsRS89CeamuExt5cTNS21nRS27ZtGygjeU0Vj73tZTQWNm5mF7TPmnuP5Dg7duzAupamiNph7zh6f9o7tZWcYE0B9vIIIYQwM7E/WEMw8oEVQgghhNAx+cAKIYQQQuiYoYZpmD9/fp+2wvQ6dBXe9AmULqCKNTGWroX80+bXN58z+cnt+j9pIkxfQn550+XYVdqWtpHf2q7umw+faNG0mVaO9DPmZ7dy0r/YnNKVZ/PVk5vYtC+mL6HxtBRRlG7FbMjSXJC92PZAbTMbojZbKooWbY+tdbrSbfZmGhVaZ6Z9obq2P5GuztaNpdAhjaC5r6jfFnLGwinQ2jG93Zo1awbKKPxHFWtSTTdomh+yLQuncP/99w+U2RqhfejAgQNY10II0Lq2uaY5Nc2m9Y/Wrz2D2mH7kL1HCBtPmmvby8i2TCtptmU2TpBd2DvHdHFGTrCmANuIQwghhHBykA+sEEIIIYSOyQdWCCGEEELHDDUO1tGjR/v8yeYDJt+3xb8w3ynFxjHNwWOPPTZQZvowKjcdifnDqb7FS6ExMp2M6UDI921tI42KaXho7G3cWtIWmOaLdAvmI6c0GVWsBWpJD2Nx12iebD7sGaRbsBAgZLOmzVu1ahWWk9aGnlvF+kX7PRpPizNkdk96K4tXRs+weHW//OUvsfySSy4ZKGvZn6x/tB4sZRdpQ6p4TVqsKapr82Sxu0inaPPUksaHyk2bZ2uHYinZeNI6s7VOekKra+8RkorQe6iK+2f2bWuS3iOWyovsYnJy8oTr2vvC7HBkZGSgzPpHdmjvC9OjtehXqX+mJWslJ1ghhBBCCB2TD6wQQgghhI7JB1YIIYQQQscMNQ7Wueee2+fLtVg1FGvIYlhZiATyW1tdindkMWkoPovFcmmJM2N5C1u0L6Y7ovrWZqprOifSxZkGwLRE5FM3fz/Nn8W7Mv0MxVcyDR1pzCyfFs2TjYXlMlu6dOlAmY096VlMt9Ki4zJ9GGmJzA5JV2NjbDZLWNuo37SmqzhuUxVrSUhHUsV2YdoQ0pfYFmxx02js7PfI5iymkOUitL2PoLGwfpCOy7QvpvmiuTYdLsWrMp0b6a1sPZnuaOXKlQNlLbpfG3fbL5YsWTJQZu840tGaHVI7TCtn8dhoXk2nSGvE9hb7DqC5sncDvQNMr0X5Ho9HTrBCCCGEEDomH1ghhBBCCB0zVBfhnDlz+o67W44o7fq/pSKga56W/oaODM1NZa4HYuHChVhObiZzR1D/zK0yMTGB5VTfjk/p2NlcXXbcTi4NO7qm6/R2rE7XfM1VYuV0HdvSctAxtaXKoSNqu6Jt16PJ7i0dDbXNXBo2f+QWMXdLiwuN7JvcJ1VuQzZ2BLkjrM82nrQXmSuIXGDmFiNbNjmAtY3cOzbXtNatbbYHkEvZ9l/aJ82tTXNqY2HPoL6Y+5lcoy2yjSq2C3OXkQvbQizQO8ckMwbZuM01tdncYhbihGQXe/fuxbr0HrHUQ7TnUKiQKt8v6BkW6sHcmoS5Ko2cYIUpxTa7EEKYSQzhLGLaYprW0E8+sEIIIYQQOiYfWCGEEEIIHTPUVDlnn312n8/e/Pqk1zHdiumR7JovQf5bS/dgGhUKv2+pKMgHbP2jsaDfquJ0H1WsqbBxGxsbGyizVCukwbEwBnYFmTRtpu0hrc327dux7ujoKJbTWDz66KNYl65579ixA+uSvoTSvVR5Cg4qt7aRDZn+wvRIpO+zkA6kJzQtCumt7LmW5oTWr2mUaIwsNIGl6yANju1P5DoyrUZLSAf7PZpXC29AGqOHHnoI65pGhcbTNJSkU1u+fDnWJXsxLZnZPV3TN70WyRVsnmg8bZ5IP1XFfTHNJvXP1qnpuCisi40b7S22RmhfsLoW7of0ZPbeonetuSQtVAv9no0n2YVpuVvJCdYUYIYTQgghhJODfGCFEEIIIXRMPrBCCCGEEDpmqBqsJ554os/XbX598pG2xPeo4pQY5kcm37nFZ6EYIaYNsdgqpH+yWEzkfrRYIOafJs2I6TIIiwdE82epDMynbnFpTrQdNqf2XBpnSvlTxSkqTDNE/v7W+GGErRHSPpjN2hqhsTN9CelR7PcIu/JuY0FrxHSVpO+ysbdnkDaHNC5VbMtm36Q9tHh8pvskW7YYRlTXxsLiAdF+ZnsZPdvWHj3XNK0GjbO1jTSQtp5oD7B+mA2RjsdiStFc23pqScVmv0c6NbNZsi3TE1u8SGqbjRvZUKvmmrC4YrTv2XpqJSdYIYQQQggdkw+sEEIIIYSOyQdWCCGEEELHDFWDddFFF/X5wC03EflvLY6S6Z9IX9CiO7K4PeSrt3605EKimEtV7Lc23YLFg6ExWr16NdYlfYj1g3z4FofFQlmQzqVFa2OaAxsjigdjv0e6o5ZUQKb3MJ0T2afl5CJNhOnDLPYT6Q4s9g/ZrGmGKKaQxW2yOEEUE8rWL2lUTDdomhHSL7bkMzSNKI2babss3hG1w/QzNCfWNoPWiNkF5Um0+G/UNnuu5XFtyUVImiZ7X5AO12zF9ISLFy8eKLM9mdavxQSjMa7ieHOm46I5sf6Rfdr+bXscaRpbdIOmi7VntMS2IhuyeWolJ1ghhBBCCB2TD6wQQgghhI6Z1RtCivCtW7fW+vXr67zzzutzV9gRpR2V0pGfHePT0bMdZ1o7yE1hx/jkZrJ+kDvQrqXa8T4d45tbhFxSVtfSXNAxrl1jpnGzFBV07dbcjGa6dOxvbgpKaWPPJTeM2ZCFG6D5s5QoZLPmQrGr6WRzLVfTDUoRZG5tc69Tvy2EBIXTsPVkbmKSFZibkWzAXAy0X9j1cesfjb25r8jdvWvXLqxrbmJaf+aqJjcMpdCq4jmhNVbl7mDaw83tR3ZvdmH7E7nnbOxpjMxNRW4m27/NxUupxuwZtIdb28wtTXZh7ziSirSkOjK3r60zWjv2rqY9riU0kO0h9p40ZuwJVle5gk4UewkSLXkPu8By2U0VLbGqWsatC2xjnA60al9eLbbZTQdea7swzeZ0oOXDtgvsj5vpgP2BPFVYPCfCPkCnCsvjOlW02IXpcKcK+8NkujNjP7BCCCGEEKYr+cAKIYQQQuiYoYZpOOecc/p8xOYjpWvlppMxdxn58O3YkY5K7Vo5XVedO3cuttmusdOxuKW5oTa3hBWoarsSvHfv3oEyu/pLfTb9hemqyHVkx9HkJrZ5svEk7Iou2Zy1bWRkZKDMdCQWQoJsyzQApPkxe7OUKLt37x4oM9ca6SdsPdHYt4R/qGJ9l2lDaJ62bduGdc3tQ/bZovcw1xPNiYXNoPmoYl2N2TeNp9ms6dFoPbRolFq0VrYW7N1ANmASBno3WNgT2vfuv/9+HGfTGJINdJFKxvY4WqsmS6A5tfVEa9X60aJTnJiYwLo0J6ZptH2E7MX2J9IZdyXzyQnWFGAbZgghhJlJyx9pIVTlAyuEEEIIoXPygRVCCCGE0DFD1WA9//zzfT73Q4cOYT3yI5sOyPQz5H83vy75Xy12kGm+yJ9tsWpIa2M+4Hnz5g2UmcaBdARVrA8yHQFpIkzDQc8w3ZHFGaH5s9RDNG6tqXla4qO1pPY488wzB8pMA2C2TO2wa+z0bLvmbfZC+hnTEpHeo0XHR2k9qjw2DtmRacxoPG2tm+aH3EGmA6L9ydYIxbejNV1VtWrVKiwnrVRL/D+ra3ZPmjZbv6TNsjVCc2qhAsxmSdtjY0/9aElHZO8W0/GRfbasJ5snG/uWsA40Ri2hFyyuYEubTUdNEhvTnppGlN6f9o6zd0YX5ARrCrBFG0IIIYSTg3xghRBCCCF0TD6wQgghhBA6ZqgarIMHD/b54S0/Emk7TKthfnLyv1o+LdJUUC60KtZ7mO7IwjeQr95yy1GcKIsdtGTJEiw3TdOJts00FeT3trEwjRmNvelnaDzNPWu5AcmOLH8X9cV0g6TBMh2JabNIG2A6J9KB2Biblog0OKYvMd0YQWNsGg7SB1bxWqUxruJxtvbankNaKfs90h2ZrmPx4sUDZbZGzC4oDlJLvKOW3JD2bLMhioNk+seWHHCmlaN90tYZzZNphmieLG6TjQXty9YPmhN7X7SsPdvLaG+wsbD3C2FjT++Mffv2YV2Kg2X6MHuHk4bS8nLS+usqRXNOsEIIIYQQOuZVfWB9+tOfrlmzZtW9995bVS/8Vbpx48YaHR2tdevW1ebNmztpZAghhBDCTOIVf2Bt3bq1/ud//qfPDXXTTTfVFVdcUTt37qyvfe1rde211+qRaAghhBDC65VXpMF65pln6mMf+1h9/etfr3e+853Hym+//fYaHx+vqqoNGzbUyMhIbd68ua666ip8zoUXXtin+zD9FPmRzV9s/ml6BuUgqmL/q+VeonZYrijTxLT8HvXDdDk2FqSrMV0G+c4tDxn56i29hMW7obhiFuuEcrWZXstixJBWxnQEpPcwbQDVtbaZ7oRswJ5BcWLsuZarjfRkts6of6ZxIL2Vtc30HgsWLBgoM7snLZnFeLLcaRSbqmX9mn3T+rXnUv7FKt4nbZ5orZoux7SV9Eey6Y6o3PRopJe0uIKm4yKdqWmlyF5GR0exLq0n0mVVuV6H5m/Pnj1Yl/ZJ0+DZGqF90p5B+5a9c6i8Je5aFWtgbewpzptpXW3/pTabxpA0WLYvtPKKTrA++clP1nXXXdf3gXLo0KE6evRo38a5bNkyNagQQgghhNcrzR9YP/zhD2vLli114403Dvzby78Eu1LihxBCCCHMJJpdhHfffXdt27bt2OnVvn376j3veU/9/d//fVW9cLz34inW5OSkhgqoeuH4+qVH9xYOn47w7Dq3fdRRubkY6FjV3G3mTqI2W2iJ/fv3D5TZFV06KrWjT3OLkXvG3EbUD3uuXemnNtvx8M6dO0+4beSOaE2JQm1rCadhIS9ojGmeq9wuduzYMVBmc03uAeuzueLJtW0hBMhdYm5mmidzU7W4L20PoPVr7g+zF3KBWigEm1eC5snSpxhkh7Z30ni2hCGp4vAUFraG1o6NG42FrSezWSo3NyO5kyxcCLlWH3/8ceyf2ay5fglaq+ZGNzccvc9sLMjFZ25tcq+by9b6TC5Qc+WSfZpMxGyZ5snWKY2b2WwrzSdYN910U+3fv78mJiZqYmKiFi1aVJs2bar3vve9dc0119Stt95aVVVbtmypAwcO1JVXXtlJQ2cStlGdjJjvPIQQZhL2h1cIRqeBRm+55Za6/vrra3R0tObOnVu33Xab/uUYQgghhPB65VV//bz0mG9kZKTuuuuuV/vIEEIIIYQZzVCPl44ePdrnT7bTLtJa0FXOKvfVk1bKfOfk+7arsea3Jq2MXW0mHYildiHt0ouhMV6OjSf126670nVVujJfxUfopssxzQhhGpX58+efcF1zVVJaBruCTBol0sNUcaoVe67ZEI2zpYYgHZdpNQzSnZhdkJ7BdBmrVq0aKDO7ME0M9c9siNaTaYZMp0ZufkvDRONmz20JTWB6FtKd2HiSfZpu1EJZ0L5lmhhqh6X9IhsyTY1pYkhvZ1pQemfYO4DWqtW13yNsTdJ+YfNkYYDonWPpnfbu3TtQZv2gsTBNcktYF9tbaP1u27bthOvas20sCAtv0UpS5UwBJkQOIYQQwslBPrBCCCGEEDomH1ghhBBCCB0zqzeEaKBbt26t9evX1znnnNOnEzL/NPm+Te9hGqwWTRTpJ0xTQdoA05GQZqiKdU6mhyAtmOkWWvJAWpoT0qLYGFO/Ta9lkDm2pOUwbYDZC2lzTItA42lzTdoA09pY7B/S9pgWhfptv2d2QVow0zi0zBPpgCjeWVXVokWLsJzSrdh6Ip2T6T1aYuGZBqdlbyFdpM2p2QXFILPUHrSPWBgZixNEdmTrjMbZ9LLUD9MXWbodGmfTfJEdms6N0qiZDtd0kZdccslAWYvmz96HZhekizM7pPlrsSFLG2RrhN5RNk+k47MQGWYXtDfY79HY27i1arNyghVCCCGE0DH5wAohhBBC6Jh8YIUQQgghdMzQ42CZBuKlkD/VfOcWh4N0Q6YPomeb/5b0BeY7N10V6U5MJ0E6Gfs902bRM3bv3o11qc2myyGdk/nkDfKpW6wpym9ldmHtIN2C+ftJ+2BjT3oGkztajCbqn+lASENnOgKLFUZxYlriylk/aE4tJo31j7QrZrMUN69VO9ESB4n6Yr9HeiuzC9NFtuQuJX2Y7WW2F5OuyvaWlphgtA9ZiBuLQUa5bs22aP7mzZuHdUkjaPlT7Rm0B9g8UZvN3kz/RDliW+Ibmr2RLq41zyntnZZLlPZ722ft92j9Wuw22g9b8kgej5xghRBCCCF0TD6wQgghhBA6ZqguwlmzZvUdP7YcD9NRdJUff9MRbEuYBrvmb8eOdLxr7g865iT3UBVfr7WUCnZUTqEFWkJImPuDjnafe+65ptQlNJ7muqBxM9equTVpLOz3KHyDHZXTGD355JPoIrB5IteaHfmTq9qOzykVVBW7bGydkTvCxrgl5Yelv6F2mKtk3759A2XmxjFJAZXb/kTuD3O3kL2Z+8NSKK1cufKEn0FX3m1vMVcllZvd07hRqqQqtiEqq/I1SXZvLvAWKQalZnryySdxP7T1RO+illAI9i4zu6B1Zu8c+j0LOUMhK6p4j7OxX7FixUCZpYOjflgqKHpuFe8jFgKG5sTChbSSE6wpoFV39HrGXronI6a/OBnpagML4bXCPo5ORrrK1fd6J7tcCCGEEELH5AMrhBBCCKFjhqrBerlf3cIYULlpAExf0HIFnZ5tvmz6vVmzZuE1T9OzEJa2gnQgLfqLKr8eS5Df2rQBdHXbjtUpZYhhqSFIG2KaONP2kGbE0i+0hKGgZ5gWxWyW9CU2blTX9EW2zihsia0R0yQSZC+WHsau/9P6m5ycxLpkAxbqw+yTyi2FEukiTTO0dOnSgTILQWDanvHx8YEys0NyxVIbqtzuKVXOoUOHsC5pOW2vpjE2SUFLCjQK3WB1TYNFa2T+/Pm4pswOSYfXkg6ude3R2rGxp3eR2Rv1zzSGpnWcmJgYKLM1SXuc7QsWTqEl7EVL+I5WcoI1BXQVQyOEEML0wP5gCcHIB1YIIYQQQsfkAyuEEEIIoWOGqsE6/fTT+66uWwwN0lS0xPeoYredxeehuhYjhPQzFl/Lfo/803aln7QrduXdXJVU39pM5dYP0p20pIGpqhobGxsoszgs5Je32CoWP4rabFo50sVRvJwqjtFk82F2v2zZsoEy076QdsXiNrVo6Fpi45iehcbYYvnYuiZ9UEsMK9NfmM6FtESmUSLbMm0X6QYpbleVa/ZonE0fRpofa5vp4mgPMB0Q7b979+494d+zeTLdGNVvGU+zWRoj0z9aDEHa1w1a66ZzsrVD7wyLbUZzajonKifdYVXVnj17sJz2SdPFki3bOrVy0h+bLpL2ats7W8kJVgghhBBCx+QDK4QQQgihY/KBFUIIIYTQMUPVYM2ZM6dPL2H+d9K+UMyeKte5kEbFtAjkc26Jz2KxmCzeDcWasjhY5Ee2OFEWB4vaZ3oW8rW35LgzXzbF1qniObF5Ik2MxZkx/zuVm0aJxs00ZjRGFpPGymn+LEUFaaJMm7d7924sJ52LjRvFYjKtHM2JzZNpiSguDbWhitevabssDyTZXEubrW2kibF4QKafIW2O7ReUt9DWr/WP2me6QXqG6S1Jy2laUIrzVsUx/cwOaT3ZvkfvItOeWhws0qmZ5ot+z3S49n6x+SMuueSSgbK//uu/xrqf+MQnBsqsz7Z+yeZa8oOa5s/WNdmF6cOoblehlnKCFUIIIYTQMfnACiGEEELomHxghRBCCCF0zFA1WIcPH+7zSZs/nDRK5m+2WC7k+zY9C+lOrG3kczY/u5WT1sJ0CxSLxXzyFreF2mE6INLxmC6HnmFtM+0SzavpuEh7ZnHQTDPSEleMYmyZjoC0AaYjsGfs2LFjoMxsiMbT7Jt0OVbfNEqktzPdArXZ6rZoO1avXo11aext/Zr+ibD1dPDgwYEys0PSStn8W8wz0giabZHdm9bV4r+RfqYlvp2tJxqjyy67DOsa995770BZS7w522fJLuydY/2jfd30b6Q9s/haBrXPdHwbNmwYKLN4ZbR2TB9mMbqof5YzkvYci8dneXVpLOzbgOra3tlKTrBCCCGEEDomH1ghhBBCCB0zVBfhBRdc0HcsbSkO6NjRrsyaO4mOZu1YnbDfoyPYZ599Fq8E23EtuQ0sNQT1oyUdQlWbi4HcFJYehlwPdqxuqWsolIU9g1w2Vteu81r4DYKOrs1lR65RCzdhLgayOZtr6re5As31RO6SlpQR5k6ktWCpNsy1RnZhx/gWAqQFcjPZWLS4P2j9kouxyt1XtHZs7Mmdb257G08a+5YQIJbyZefOnQNl1113HdYll3RV1S9+8YsTbhutB5M7WNgLCgFiv0d2by5pCjnT4k60ttl6ojBHFvqI3r/mhrX3CNW3sEX0HWDryUIU0Xowd+KuXbsGykZHR7FuKznBmgIs/lQIIYSZieXqC8HIB1YIIYQQQsfkAyuEEEIIoWOGqsE69dRT+3zEdl2ZtDbmAzafM+mOTLdAzzZ9AulkTjnlFPTLm1+f+m3XUkn/ZFem7Rlr164dKDNtD2kGTCtHV3dNn2A6GdJaWP9IM2BX0G0saK5NK0Vjb9eVyd5MR2BjT7oxs2/TCBItGqyW9BIPPPAA1qU5tbGwNUK6I9PVkS7Owj+YfVJYCKtLOh7bn0hvZVor258mJiYGykyXQ6FTTPNl19ipHaafoRQsluaGdC5vfetbse727duxnGzZdJVmn4Tt64T9HklFbIxJF2fjZnsn6TBNu/ahD31ooOw///M/sS6Fp7GwPqbPJeydSmvPxt60vPRsc/GSbszsrZWcYE0BthGHEEII4eQgXwIhhBBCCB2TD6wQQgghhI4ZqgbryJEjfXqJlvD0Fm+DtC9VrHOyGCikZ7A0EpRewGIxmdaC/P0UF6WK0xaY7sj81tQ+0znReJqffXx8/ITrtoy9pTmhNpP/vsrnhPQoFreHNBEWU4j6YdonGwvST1jcLdL8WKwpmxOKYWRQO6xtpBuz9WTpQcjtbhos+j2zC9PPbNu2baDM5ppi1tl6Is2I6ZlsD6BnmH2TDVnsINP2kHapZW+xfY+ea9o8GyOKFWXxnEhLZPGOKPaTvVtMV0X7hYXwoWebbdr80Z7z+f/3/7AuPfu//uu/sC69n2ydmtaVbMv0U7RP2jvOxp7s3rTDtP8uW7YM67aSE6wQQgghhI7JB1YIIYQQQsfkAyuEEEIIoWOGqsE6/fTT+2JTmD+c9CUWCsG0AaQPMZ0E+YvNB0yxNSx+jWlUCNPJkE7NYjGZdol80RZfiXz1FuNn+fLlA2WmI7GcXIT1g9pB+c2qqlatWoXl5MO3mEmkDzHtC7XNcmGZpmJsbGygbPHixViXNCoWG+tXv/oVllN8NMrTVcW6mpZ5Mps1u6A2m/6C4piZfdMYV/EeYLnhqM2mJ6V1bfGOTJtHbbO9hdrckjOyirUyNtekqzK93WWXXTZQZjZra4TG3mKpkS7H1jq12TRRtp7oXWT5LOldZHNqdkHzdJ7o30iv9b//+79Yl9aTjYXtcWQXLWvEYgVaPC7aD239kl2Y5q+VnGCFEEIIIXRMPrBCCCGEEDpmqC7CgwcP9rn67DovXcc294BdpaVjdbuWTNgxPrkI7fq4He2S+9GO4Ok42q6PWzvoWrkdwdNRsLnFyO0zb948PAq242EKp2Gpa+h42NwR5qpscaFQO8wlTc+1VB3mcl20aNFAmfWP2kxXzavc/UxXrFesWIF1qS+WloPCRVBIjyoPv0LPsLrkZrJUMjTGVexGt/6Rm8nCQpCrw/YFm2uye7vyTvZpbhVb17R27PfIDi38x/r16wfKbP9uCVlh8pGWObV3A81VSwo0gyQM5iK0vZP6ba54emdYPywlGb37zJbp96wu2YCtBZtrkgSY+5Lm2tZCKznBClNKS2yzEEKYrtgHwcmI5fUL/eQDK4QQQgihY/KBFUIIIYTQMUPVYL38WrD5Wc13Spi+gHz4pssg3ZFdH7bro+TbN01My5V+Opq1q+3WZtJlmBaB0kuYVo70F6Zds1QEpHOy/pFmyPRodpWadBKm72sJhUCaNtM+me6E3Kum16LUFaYBsWdQO0yPZmuVID2aXVdvubptmjayZdMjGqTvsmfQHmAhD2jsV65ciXXNJUX7iOkUSYtic9pyFd60PZRiyFJPrVu3bqDMtE+W5oTKLdwPzamtX7JZSotW5SFA6P1i1/9prk2DZamA/uRP/mSg7E1vehPW/fa3v43lJ4rtIQb1z+yb9IuW9svGiPYX21vIBuz3WskJ1hRgHyshhBBCODnIB1YIIYQQQsfkAyuEEEIIoWOGqsEaGRnp03KYFoW0D6btMW0A6Y5a0rVYvCPyv7fEsLJ2mG+Z9BOkRary9D4U48O0a9QO852Tzs30Hpa2gDRUFgOH2mYaDtMMkK+edGdV7Je33yMdl8VsMT0T2f2aNWuwLulnTJdjdkGpjiYmJrAuzavZEK1J077YXNP82dgfPHhwoMzWpK0zWiPWZophZDGFSEtm6VpM60ixn0xPSnNt82/QmjSNGUHzUcV7p8Uf+vnPf47lFMfMxp76YeNGz2jVUJIu0n6P9lRKJVRV9bu/+7tY/gd/8Acn9Nyqqq9+9asDZTb2VG7vX5PH0DMs1APtIxan0bSctHfSfFRxmqKWGGbHIydYIYQQQggdkw+sEEIIIYSOyQdWCCGEEELHDFWDdcYZZ/T5Yc0/3ZKL0DQc5FNt0XGZnoWwGE+Wv4s0WJa/i2LKWBwWi9vT4p8mf7jpmUhLZD5502WQr9387KSLMz2E+fvp90xrQxoO0/DQGJnWyp5B/bM5pfG0+EOmlSJtTkucINNwkN2bpsLimJHO0MaTym2/sNg4lK9zz549WJf0b7aeyD5Nm2dQ/C+K3VfFsZ+szy1jYXs17X1mFxTPy9aC6ZHIjszuDx06NFB26aWXYl3SOpouZ/Xq1VhO8Zx+53d+B+vSsy32l+kiH4b5GxE7JN2f7ev07rMcnhYrjOzC+kfv35a4iVVsA7Z30thbvLJWcoIVQgghhNAx+cAKIYQQQuiYWT3LjzCFbN26tdavX19nn3123/Xp1jQndJRoYRrI/WHH6jQk5npsaZsdf9Mxvk2LuRnJZWNX08mluH37dqxL7gELsWBtpivdNvbkQjEXg6WMIPeFuWHoSNuOv8mlbP2wI2ZyHZlrhuyiJdxEFdu9Xd2mq+l2NE9Xm81lZ/ZCa6TF7u06vrnMaYwsfRW5GGxOybZsvyC3URWPnbnRyYasHxYmhUKOWFiIXbt2DZSZ65hcsTb/n/jEJ7D8r/7qrwbKzKX83//93wNlF85nWcKBBwaf8eEPfxjr3nvvvVhO651cj1Xc73vuuQfrUj+q2MbHxsaw7je/+c2BMkuVQ/ZtbnTah6rYhswuyCVpYZJsL6Py8fFxrEvrydzatJ5s3dh70pixJ1j2ITUdeK3bZhvYdKAlXk4X2CKaDpguZ6pojXf0WvJarxHbtKcDLXkdu6CrPGtTAX1cTRfsj6mpwv6AmA681jY0ndfv8ZixH1ghhBBCCNOVfGCFEEIIIXTMUMM0zJs3r+/6vKV8IdeK1TUdAWl7SEdS1eZmMr0HpVUwTQX58M0fTjoQ01+Y75yum5ueifRB5pIcGRkZKLOQAObjptAEpsGiK7p2Fbfl96x/pJ8xPRPZ4Y4dO7CuXSunebW5pt+zfixevBjLSSNooQlo3MwdTCEyzC1ma49cEqZnIv2T/R5pDI0Wzaa5NMhmW9ZTFdu4PcN0mITtqZSSyMKeUP9Ma/PP//zPA2WmR7zkkkuwnPat/feza400lN/+9rex7re+9S0s/9GPfjRQtnPnTqxLtmXvHFq/5iL84z/+Yyz/rd/6rYGy7373u1iX5sT2Flp7ppU0jShpKE2/3JKezfYyarPZLOm4unIH5wRrCrCcVSGEEGYm9HEVwvF4RR9YzzzzTH384x+v0dHReuMb31jXXXddVb3wF9DGjRtrdHS01q1bV5s3b+60sSGEEEIIM4FX5CK86aabavbs2bVjx46aNWvWscjCN910U11xxRX17//+77Vly5a6+uqra/fu3ehOCCGEEEJ4vdL85fPEE0/U1772tdq3b98xX+mLGpLbb7/9mD9zw4YNNTIyUps3b66rrroKn/XYY4/16TbMP02+U9MnWIoZ8qlazI6WeDnkW37yySebUomQz9nSMpCOwPQepkeiOE+mfSHtkunD6BmmfbLfo76YtofKzS7sWjFpHyxmFukLTFdHKYKWLFmCdW2M7rvvvoEy0xxQPC/T4Jn2gbSOpuEhO7T1ROva2mYaSvo9i0tFeg1be6QvqmItkNkWpS4x7SG1zcbY4ua1hGWhdWa/ZzoXGyOCNFEUR6uKw4j867/+K9a18aT919KnUL9btGT2frK9msbe9mTSCNv8m2bzQVgjmzZtwrrUb4v/R/22+WiJg2X9ILuwd4BpK2m9W2w62pe7CpHR7CLcvXt3zZs3rz7zmc/UW9/61vrt3/7t+t73vleHDh2qo0eP9r1Yli1bpiLZ1zPTOS5VCCGEEKae5g+sI0eO1NjYWF166aX14x//uL70pS/Vhz/84XruuecGvriHECQ+hBBCCGHoNH9gLV26tGbPnl3XXnttVVVdfvnltXz58mPujJce5U9OTqpbJIQQQgjh9UqzBuuCCy6od73rXbVp06Z63/veV5OTkzU+Pl5r1qypa665pm699da6+eaba8uWLXXgwIG68sor/cdPOaUvZYblIiSXm52OWe408r/ff//9WJf0M6YZIX+x+fVN+0D+ZdMRkA/fUrBYXJuWGCEU58u0GuT3bo2XQvon0z5Qv1+8cPFyTOdEGiq7lEHpXVriNtn8W5ygtWvXYjlB/WvVKdIzWrQPplsgjYppUUxXRfVN70E6LtOM2O/RnmO2Rf2z+HiUU68lX6Bh2jWKsWcx72w8aSxsPEnnYtoewsaNNEpVvDfYmiSbtb2M+kzvhSq3ZXq2zSmNp/2exW47Cr9neQupHS2ptVrGrYr1VrYfkh3a75G2y55t73B653QVaukVXe/7yle+Uh/96EfrL//yL2vOnDn11a9+tS6++OK65ZZb6vrrr6/R0dGaO3du3XbbbblBGEIIIYSTjlf09bNixYr6wQ9+MFA+MjJSd91116ttUwghhBDCjCaR3EMIIYQQOmao/rvHHnusz1dqWhTSOLTkSKtiH77pg0gnYXnP6Lkt+emqWINhegjSF1iMH4uZRHFpTEdA/TadBOlZLEYX6Zmq2IdveaHIz24xaUxvR5o9i5dCc2K6BWqHxWyx3yPMvkm/aHoIGyOqb7pI0ltZ3B7SPpjN2toh/RPFQKpqyyVqMc9Ig2F6NOp3i/bQ7ML0OtRm+z26ZLRv3z6sa/sTaTlNo0Lr2vYWsjebO5vrlphJtF+YBo9+z3Sq1j9qm2nXaJ+1NWJx7KjfpoukdW2aKNLxmYbW3g1UbuNGceWWL1/e9Hu0Hmx/ovVg7/tWcoIVQgghhNAx+cAKIYQQQuiYoboI586d2+fysNAE5Jqxq78tKRUszQkdn5rrgpgzZw4eD9sxPrkC7HiY3C3mbqOj1iq+pm3PoKNgc7fREfrDDz+Mx/521ZiuXZuLkMbN6ppLg57R4iK0Y3U6mrfjbLuOTc8wFwq5jaxtluqIbMBsiFyr5rogGzL3pbmp6Jq3hU0gt725dyyUBe0NJmGg/plrlfpnLkJzX7ZICmjfsuvq5vYhzPVE69qeS/uQ7d/mLqVn23qi9UApkarYvXfkyBGcK2szzbXZG60dsyF7F5Fb6/LLL8e6lJKoJUySpbmxvZPcl+R6rOJ52rFjB9Y11zHNk+1PRFfZWHKCNQW0xBN5vdOihwkhhOmKfQiHYOQDK4QQQgihY/KBFUIIIYTQMUPVYJ1zzjl9uhTThtB1zpbrtVWswTB/OGlRTBPVErLAtAjkOzd/MbXNQlaYLoOuq9rV5pb0KTRPpomyNpPOxfz65Ks33YrpFqh9pg1oeS6l8TF3qWkBSQdg2jV6toXIsOvKpCUxHRfpCW2uSVtpOhnTxJAdmqaR9HY2FpaChcbIQmSQzVqaKrIXG2PTRdJcb9u2DeuOjo4OlNkaMf0bjYWNG2mzzLVGe7Jp5UwTQ+8MG08aN6tL82R7vT2D9KS2X5Au7uGHH8a6tnb27t07UGbvVJKxWAotCuti+kezIZo/s8PVq1cPlNlYWP/o/WvaQ3q2jXErOcGaAuzjL4QQQggnB/nACiGEEELomHxghRBCCCF0zFA1WEeOHOnz75ufnXQZpHGp4jQ3Vewnt98j3QKll6lifYLpZMxvTdol82WT7sT8xRZXjDRUpn0gfZDp0UjjYDow08SQbsF851TXtCHWZhpPi7FGsXhs7Emv89BDD2FdC+tBGkFLL0FzahoHewbpQ0yvRXoPS830hje8YaDM1qm1jWjR65i2y9Yq2YWt35Y9gOzQtJkta8R0QBSzzuJ5mX1SmhLT21HaEYuvRP02faeld2qJ3UU2QLG4qnjcTBPXYltWl/Zq02zae2vFihUDZbT2qrjf1jbaw20sTANNOifSWlXxWJh2bXJyEstpL2rZ9yydVCs5wQohhBBC6Jh8YIUQQgghdEw+sEIIIYQQOmaoGqxf/epXffqFJUuWYD3SQ5gfmuLlVLFOwvQz5C823zJpAyw+D2kZqliPYrGRqH82Fi0xjCy0BOmfrG0UR8l0QBZzhXLcWT9I22HxwyjWWBXH2DKtDc21xcahWFqm97D+ke7PdE5ks6afsjVCmh8bT9LxmC6DtA+mLzHNCGnaLC4VxV2ysbBymlfSuFTxejJtUEssJmsbrUnTbJKWyNaCzQlpVKx/pNcxvR2tddO+WL6/Fm0P2bc9l3SRpuGx/Z70Qa35QQmL00djZ1o5aluLttZ0fKZVprVj9kbr1/psa5Liwtm+R1pX0wK2khOsEEIIIYSOyQdWCCGEEELHDNVFuHDhwr6r1nZcS0fadmRoR+V0pN1yVdxcXZYGgo7x7VidXAGWxofqtqbxoevtVpdS2lx88cVYl6557927F4/m7WiXUoyYK5fcjGvWrMG65kaltlloCRojS+Nj18rJ/WjX2MmFYiEdKLSErRELb0D2afZNY2S/Z+7nlnQU5JIYGxvDutQ/W782ni1pisidYO49khq0uEqqqrZv3z5QZuuX9hHbL2yM6Pq+ubrIDUPhTey5thZsPMmGWkI32DvAQrWMj48PlJnrn2hJXWPr5t5778Xy3//93x8oszVGa8T2yBZ5hUlpSFZi9kZts1A9Fr6BQi2ZDZEb3dZeKznBmgLMGE5GLJbPyYhpu05Gusr1FcJrBX1cnazYH5ahn3xghRBCCCF0TD6wQgghhBA6ZlbP7phOIVu3bq3169fX/Pnz+3yddu2afMPmLzb/O/n2zV9MGgW6DlzlehZKZ2HX2KkvplEin7ONhWmJSFdl2gfqR4sWhX6ryjUj5PtuuTJrbbMjbWqHLQkqt/6RhsNsxX6PNAqWCoj0b6YNMTuk9DemBSTdkKVmWrhw4UAZ2VWVXysn/YTtF6TNMk2F6ZFormyeKLSEuYNJa2PhaewZNH+2l9F+aPNkV+9p7Vg4DdLRmoaHbNk0tKbjIr2s6XKof7a3mL3QXmvX/8lerG30e2bfltKI5sSeQXZoWkBaeyb9IA1tFdu4aTZb0qVZOyhkhYXvoH7TXljlOnEjJ1hTgL08QgghzEzsD9kQjHxghRBCCCF0TD6wQgghhBA6ZqgarEWLFvVpHUwbQEez5pOn2EFV7PumOCxVrEcynzz5sk1rY/5p8p2bNoB8zqYjMQ0HaSLM/046oP3795/wcw3TjNgYETSnNvYWc4Vsy/Ql1Db7PdIumYvBxp78/bZGSEtmcV8MsnuKEVPFGjPbSmid2Rhb7B/SZtmaJG2Ptc20gKQ7slQpFHLCxp7iR1l6GFtP9AxbN7Q/mY7E0jBRv23saX+ycWtJR2Prl/Y+0xhSXUsFRRpYWwumA6Jnm16LxtP0j6YHpjVFKXGsrs0p6eJaUhdV8d43f/58rEtr0lKr2XiSZstinpHdWz9a5T85wQohhBBC6Jh8YIUQQgghdEw+sEIIIYQQOmaouQiff/75Pl+u+d/JB2waLNM+kF6nNaYFQToni2tjuiPSF5gGi9I1WMysllhaFgOH9CH2XPKTr1q1Cuta/CjCtCH0ey1x0Kq436aVIo2C6d/Ih28xaWzsSY+0cuVKrEsxZczeLI4OacGszWQXpqui/lmf7fdIe2Zx3qgdExMTWNdsmezTxp4wnSLNielGTfNDNm56FtKdWJ490xPSXmvaJdLP2L5O+hlLoWT7Pf3ejh07sC5huUvp3WDrqSWmm80TvQNsnZoejfZ1q0tjbzpFehdZDk+bpxa9LI297Rf2riUNpdkWjVGrftXICVYIIYQQQsfkAyuEEEIIoWPygRVCCCGE0DHTKheh+ZypiZZnzTQxVG5+ZPLJWmwV0kOY5sD8uvPmzTvhuuRbtpgdlr+Jxpli61RxjBfTOZG2w+I2WfwSmifLv3fOOedgOWHaANJV2ZIgG7BxIxsyezPdEY3Rrl27sG5LXCrTjJDdWowu0i5Z3DXSv1ncppbYOLbOyIYsxo/FwaK5MjukNWlrxHRqhGl+yJZtnkinZm0z3SfF4zJ9WEv/SDdo68m0cjTXpsEiTZTtIaT5If1rFe/fVTyeZm80T/YOoFybVawxshhdpHOy3ID0HrEYkgbprSzvaIvO2PZUsguzWdJA2t5p8biMnGCFEEIIIXRMPrBCCCGEEDpmqGEaTjvttL6jQ7tGSS4wc0fYsSo9uyVMgx21kivojDPOQHeSuTTouNXcRvRcC0Fgx6d05G/Xrildgz2Xrraba9XcLUuXLh0os+NaGntzBZlLilwB5mamcCHmbiH3nl1LtrGgOTEXAz3b+mEpOOgqdYsLzVxahLmIze5pvdsVdHIztbj4q9gNZ+4rsgtzlZEt2xV0mz9yu9v+RP2mea7ysBd79+4dKLPr8RTixPZZeobNqYV1oXky1xPtewcPHsS6NCfLly/HsWtJJ2VjT2E9bN+z9yStEXs30JyYfVM7zDatfy2pw2jPsf374osvxnLaU61/5HK157aSE6wpwDbzEEIIMxP7eAjByAdWCCGEEELH5AMrhBBCCKFjhqrBeuqpp/p8waYjII2DXSm26/iUdsQ0MVTXQiGsXbsWywnTs1DYBGpDVZuGo+VKsOmqyFdvOonR0dGBMrs+3qKrst8j/cTChQuxLtlQFWt+WtI9mOaLtD02H6YDIo2ChTGgcbZwDNY/0nZYyheyIZtT0h5an00nQemS7rvvPqxL42xjb1oS0ohNTk5iXUq3YvsTzamtdRsj0odYKiCye1tPlP6oijVNLWE2TD9FNmS2aeuX6psdtmiUaDxt7Vnbdu7cOVBm+rCWfpg+iPpiv0fhBkzbRbpm0spWeVge2gNs7ZFdmGbT3LZk4xbqgd6/FHrllZATrBBCCCGEjskHVgghhBBCx+QDK4QQQgihY4aqwTrzzDP7dFCmc6KYFhZbxfzW5Ne1+CXk7zX/NGktTH9hOhBK19ASI8R88jZGFHfL+kfaANNUkL/fdHUtvnOLbUZzapoo+z2K3WTpM0jDYTo+aptpX0yLQOmgTItAa8eea7oMGmfTtBEWS420K9Y203xR/yyeF429aSqsHbSPWOwnSzFCkF7SwrqYHVJfSP9Y5boxwmyZxsjWJOnJSD9XxevJ9KS2rsmWbd+jfbIlfUpLarUqnhP7PdI0WToa0+bRnNh6orVj63f58uUDZaYbtP2J5sTeh7SezC4svR6NkcXpI92Y6e1ayQlWCCGEEELH5AMrhBBCCKFj8oEVQgghhNAxQ9VgPfjgg33+a9PJkD7ItD2mhyAtkelLyH9r8T3It2z6BNNaUJvnzZuHdckXvWvXrqbfIy2B+fUptorFEyHdWEuutyrWz1h+OorzZLoV8/fT/JkWgTC9HeWis+eef/75WE6xeKwftB5Mt2Jxe2j+TF9C/Ta7pxhdppOh/G32bNNfkH235B2tYi3JgQMHsC7ZuOkiqW0WK850mDQWNvak46PcglW+/1K5aWJoz6GYS1VVK1euHCizfK2moSNbtnGjPcd0ObQnm3bR1tmOHTsGykzbQ22zHKX27iNdlO2dVE62UsX6R9uzWrD3E+1xps+2Z5CmjXIOVrlGsAtyghVCCCGE0DH5wAohhBBC6JihugjnzJnTlxrD3B+UPsOuzNoz6MjfjubpKNiOa8mlVcWuFXNT0HGtPZeOSukabZWnsyA3ml27pXQ0Lf2oYhev9Y/cETZPdExtLmL7PXL9tqTVMbcRzZM911LakJux5Yq2uXzMrUluH0ufQi6wltQ8ZrMWZoVswNzB5GYyl7u5QGk92PxRXbNZGgtz71nbaJ7M5U6uNXLNVflcU7/NZUNtXrVqFdalMAS2Ti2MDLndbdxoPdD+VsVhGp577jncU2mdVnFfzC4Ic3+ZvZCcw0I90Bqx/Zveh0ePHsXwHbbnkNvW3uFmA4TNH4UcsbbRGNke2UpOsKYAW+AnI7YxhhDCTML+YD0ZsfhxoZ98CYQQQgghdEw+sEIIIYQQOmaoGqzTTjutz89srjVKqWD+abtqSv5e03uQbsFSAJgfmXy4pCWrYh+3aSpIg2P6sIsuugjLSXdE/vsqHnsLb0HHxhaawNpG82rPoCv9Nm42RqTjMb8+XbG29CKkD7O0JXbNm9pmWiKyZUuTYXZI179bdE4GjZGNhekvSHfUoou0a/4WboB0VbZGSI9mY0x2YTo+08TQnFgYGdqHTItia4f6Z3ZB82prhMbY9EymXaJ92fZqWmc2/2QvtvZa0gaZHZJ22PZZe+eQLs7ehxTixPSd1DabU9tzSCpi65fs3ubftJzUDqtLNmupp1rJCdYU0JVALoQQQggzk3xghRBCCCF0TD6wQgghhBA6ZqgarNmzZ/fprizFAflvzc++Z88eLCe9jv1ei/6iJSSDaSqIFi2K6VlMz0AuTGvb6tWrrYkDWEoMwmImkc7JUlRQiiBLUWLPID2KpaKgsbd0LaQvsBhtlq7D4i4RpMGxsTDtIY2n6UtIj3LppZdiXYozY/oSW5Mt6YtobzB9idkh7RcWU4hsy+Kxkb0sWbIE67ZoRG0fonLTAVn6E9LbWdsojZbNHaUzMV2OrREaT3sG2YVpiajPpgOydwPVtzGmPdm0mZRCq4rjm1l6mBZ9J+libYzNDukZphukui1x0KpYb2VhNsgOLc5bKznBCiGEEELomHxghRBCCCF0TD6wQgghhBA6ZqgarMOHD/dpPEwnQ35d02qQdqKKNRGmtSHtimmUWnz1RksupMWLFw+UmX/adCekO7D+ka/dnkv6KeuHpVogDZ3FfaGxsLqmJSINVstcG6RRMa2N2UtLXDHSl5jmwMaedC72e6STsPFp0QzZ/FH+S4vFRL9n+4Xp3EgfYtqlljyJpMOzMTbdEY296fvIvq3PpjFrielH69fqknbU9jKzWdqLbK4pv6Bp5UiHa5pGswt6v7Roea3PNkYUx8x0VfRs2/doTk1LZm2m2II2njSn999/P9a1OGbbtm074d8jGzCbbSUnWCGEEEIIHfOKPrA2bdpU69evr7e85S21bt26+sd//MeqeuEv5o0bN9bo6GitW7euNm/e3GljQwghhBBmArN6dt9W6PV6dcEFF9T3v//9uuyyy2piYqLWrl1bDz30UP35n/95LVmypG6++ebasmVLXX311bV79+6Bo8etW7fW+vXra/78+X1XUe1qpKXFIfeVuRnpyqulqKD0AubSsOGjo0tzl9Hxpx192vEwtc9cNuZOIMjFYFd07ffoyNeOys1NQdgY0VxbyIqWdBbkejB3i40xuY7MTUHH7XY0b1e3aT2Qu83KLdUGubVsLMxVSaEabA+g+bN9wcKWWIoYglI5mQuF0nKYq8TcxNRvW2dk37b26Aq61TdXNa1f22fpuTbutgfQWNA6rWI3qoU3aBljCjdRxWvE9nWyF3OB215Ga6clnIa5qmmubZ5MXkHvHBt7Wg/2TjX3HI297XvkDrb+0TzZetq3bx+WG69Yg/Vi/JhHH3205s2bV6eddlrdfvvtNT4+XlVVGzZsqJGRkdq8eXNdddVVr/RnFFsY0wFbLFNFSyyu1xrTa00VtsCnA7bZTRX2EpwOWBysqaLl4+q1pquYOydKi5bwtea1HosW7A+TqcL+MJkOvNbvnNd67Lui+QNr1qxZdfvtt9cHP/jBOuuss+rw4cN1xx131GOPPVZHjx7tO2FZtmyZBv4MIYQQQni90vwZ+txzz9XnPve5uvPOO2tycrK+973v1Q033FBVg0eJjd7HEEIIIYTXBc0nWD/96U9r//799ba3va2qXnAFLliwoH7+859X1QupC148xZqcnFTNQdULftmXHt3bMSA9w3QrppMgDYfpJEgTZdonO9ImH3BLGh8bC9K+mFvM0rjQ1d2WNBCmtSE3DF0drvI0CTQW1g/qtz3Xxoi0Ujb2ZAOm1yJ7s7otOgljZGRkoIyuRle5hoPKzRXf4gantE/2XBt7cmuZzo2eYS5CCvVRxXZkNkTPsBQlpA+z+Tc3DIVDMduiZ1garpZn2DyRxsjC4ZCuytzaLZpN22fJXtasWYN1KW1UFadgMTukNEz2ftq7d+9AmaUpM9chjaft1STdMK0V2YVp+2zsqd+miyUNluncbE3Serd5on2yK0lB8wnW4sWLa9++fbV9+/aqesEQd+/eXatXr65rrrmmbr311qqq2rJlSx04cKCuvPLKTho6k6CPqxBCCDMX+rgK4Xg0n2CNjIzU3/3d39XVV19ds2fPrl6vV1/+8pdr4cKFdcstt9T1119fo6OjNXfu3LrtttuaEhyHEEIIIbweeEVfPx/5yEfqIx/5yED5yMhI3XXXXa+6USGEEEIIM5nmOFhd8GIcrIsuuqjPL/rAAw9gffINmz6F9AlVrBkxHQGlHTE9i53QkTarRUdA6WyqOD6P6ZxMJ0M6CXsG6SdMG9Jy/ds0OC1pkWgsLDaSxdGh+jYWhI1FS6oN043R/LVoA8yGLBZPS7wjiiFnekvSNJJmrMrTxtBcmxaF1qSlnTFtB7WD9oUq1oyYRpT2MtOiWJoTshezC7IBs7eWGIKkL6risW/ZW4wWnZrNKWk5TY9GbTPNrpXTXNs7pyW8jP0eaYxsT6b5a9EjWvgd26tp/swOSQtq82T7L+1PtIdU8TyZHtFiZxrTN4DSDGY6x3IJIYQQwtSTD6wQQgghhI7JB1YIIYQQQscM9YrfI488ckIh90lfYHE4TFJGmhjTAJC+y/QspDmwOEOWZ4+0D6aTIR+w+YvNr0/aHtN7kK+9RS9gmijTxJA/3DQcZBc2Fi25zOz3aJ7Mflv0JaaVa9GXmHaJuPTSS7GcdBmmtaB5NV0O6a1Mf2F6pJ07dw6Umc2SzqVVp0h7gOkGaS+yPYD2C4vP06LjMk1qy95ic0K2bNpTWn+2X9CcWB46G3vS9ljbaK2bJorG0+JEWfmrvUFve6dplakvK1euxLqkaTJ7ozE2mzW7J92Y6aVJs9maGo9+z3SKLbHbWskJVgghhBBCx+QDK4QQQgihY/KBFUIIIYTQMUPVYJ1++ul9fmrzWVMsD/PVt+QmsnAK1A57Lml7LMaPxXIhLYJpRsinbrqjlhxglkOKNDGmWyANR4u+qIrnxPRaNJ4W92V0dBTLSc9gOiDqi2miSPtiMWJM30c6Bxs309UQExMTWE5jZzZLc2JxqUijYjGlrB+kt7LYOKThsBh0pjGjmDm2P9E8WbwcarNpbUxPSjoX2wNoTsy+LW8d5cmz3KwLFy4cKNu/fz/WpfG0vdP0QS1x4VpiI9G6tnGzNtDv2XqiGF0LFizAusaKFSsGylr0dqZpJPbt24flJ6KpfhHTYNG7z9pmY0/vOHvf0zqLBiuEEEIIYZqSD6wQQgghhI4Zqovw3HPP7XOP2HE9pamxq7F2fZSOv+1qMx0Z2lErHdcfOXIE69sx96sN9WDuS7vaSkew5rKhsbBjbrqm//jjj6PryVx2LUfMNBYWCqPlyNdcMzSeNqfUP3Nrm22R28fsno78zS7M9UCuXwtvQf22I3+6Mm3r1NpsLmyCXLm21m3PIRdvy/5k7ktyVdr821iQ68nc9uRyszViLl7aG8zdTe2wvYwwl4+V0+9Z22hd2/olt/3hw4fRxm3Pon5bXdovTO5grkrC1i/t4SZ3oLVjLncbe3rHWbgJWnvWZ5u/ZcuWDZTt2LED69KctNjs8cgJ1hTQood5vdOSnzCEEKYr9gdECEY+sEIIIYQQOiYfWCGEEEIIHTNUDdbzzz/f50O3K/10ldb0Caa12bZt20DZmjVrsC75re0KM+kyZs+e3eS3Jk2U+adpLCzFhYV6uPjiiwfK7Fo5XcfftWvXCdel36pyNyr5w83/TmNsKZTGxsawnCD/fRVromxO6dq16acMsnFLiUP2aTon0yORTsLmiTQx9nukzTOthunDqH/WNtKSmBbFxoI0VNY/sgHTRJHL3EIsmN2TfsbGk/RWpgMyuyftoWk2aS8yu6e909aT6cYoNIG9R+j3SO9TVTU5OTlQ1hKmo4rXiI09jbHpziysB+05FgKGQmfY2NM+1JKOqIp1mLZ+6R1uUhPTjdH8tdi36WVbyQnWFGCbdgghhBBODvKBFUIIIYTQMfnACiGEEELomFk9C/ozhWzdurXWr19fF154YZ9/12LHkJ7BfMCmO6Jnky6rin3c5EM2TANgWgTy91ocDtLg2O+Zr5583zb25Ktv8bNTmo0qj4NF82cmSr9nY2zlpImweECkRTANB7mJTbdiKRzILsy+SQdiuiNLO0E2YGOxaNGigTLTQ5D2xfQXtq5JH2JxqWiMTJtpv0f2Yv2jui1pg2ytmw6TdDW2JqnfpE+paksnZGNBuiGLm0d2aBola3NLWjP6PbND2ltI41Tl80e6QdPL0lyvXr0a65o+iNa77RcUH420klWsf7KQFZYWid5FZm80J6ahtdRKNPaWIoyeYamAWr4DqnKCFUIIIYTQOfnACiGEEELomHxghRBCCCF0zFDjYB0+fLjP527+Tco3ZHGpTFNB/mmLi0G+dnsu+YtND9GSi87GguLBmKbG2kwahfHxcaxL/nd7LuknbJ4sxgv9nj2DdCAWO8i0CC1pjSgGmc2T+fCJtWvXYjnluGvJUWl2Yfm7aP4sjg5pzKi99gyzIYsJZXoNgmzIxsJiui1evHigzMKv0Lo2TSPVNa2cxXSj8bS8oxTfzmzIdGqkwbLxJL2kxW5riWNnOq6WmEm7d+8eKDMND2noli9fjnUtBhn123SYpGm0/JsU+6uq6v777x8oM63Uzp07B8psjMk+bZ813e+FF144UGZ6O7ILW0+mtyOtm9kW7UUt+83xyAlWCCGEEELH5AMrhBBCCKFjhuoiXLRoUd8Rr11tpmNOC01g7g+6xm5Hu+SSsmv+5Aqy59rRPLkN7Kh15cqVA2V2fdiuBNMzLKUNtcPcEdSP008/HZ9h7hY6mrXrw+QWM5eWQUf25jakckuJYu0guzWbJewYn36PXAZV7gqisTBbpvAbCxcuxLp2vP/LX/5yoMzssCWVCM2Jhfog10UV25a1jdafualoniw9ELm0qtiGLF0Ltdls1iC7sL2F+mdjT64ZcyeaK6jFNU5uX3vnUJvHxsbQlWfvBrItc6PTeC5ZsgTrml2Q264ljY+FnKH94tlnn20Kv0JuRkt/RPNkrlXbO+k9YqFFSObRukaMnGCFKcU+FE9G7I+CkxH6uAphOkMfVycrFoMu9JMPrBBCCCGEjskHVgghhBBCxwxVg3Xo0KE+P7xpbSicgvmh7QorYfoSKjc/u/nJyaduV9PJ922+evKHU/iAKtd8kX7GrqVS6gPzh1PbTCdj2gDyy5tdkDbP/PoWIoPG3rREFOrBjsrJnWBaFNOX0LNNu0ZjZCEk7Go6XZs2tybpjuz3KEWF6Y7M7mmuLfQGrTNbv2bL1D8LIUFj1GJvppWzdFKkO7F+0F5mGjybP9JVWWgJsi3b92g9mPbF9mpaI6ZTJBuwvZPG+JFHHsE2m96O7NPWOu3VNh+2d5Jtmd1TP2ye6F1mbbPQC9RvaxvZgI2xtaNFV0V7p4VaaiUnWFOACUBDCCHMTIaQtjfMcPKBFUIIIYTQMfnACiGEEELomKFqsM4999w+/74dwZIuw2LrPPzww1hOMTAsrs0DDzwwUNaSGsBCE1ibSTdksabIl23+6ZZYHuY7X7Vq1UCZ6ZxIB2JaDdMRUGqPFs2QaRxadBIWL4V0NTZuVNf0WlZONmdzSrHbLK6NrZEW3QE922yW6toY29ohvYZpOEhPaG57K2/R95EGy+KgUdw82/esf6ShmpycxLqkX7Xfo7VXxTpTC1lg2iWC1mpLup4qHnuLxUTPtueSzdras3cD6YAs7ReNve2RpoukvthckwbSxo20h6YPJM2uPdvWOu1PpsGzOHa0L5tei+y7qzAUOcEKIYQQQuiYfGCFEEIIIXRMPrBCCCGEEDpmqBqsgwcP9vlKLYYV+fBN22OxcSi20fj4ONaldphWg7Q9ptWwOCMUA6Ulh5T5zk13RPXtGZQbzHQE1D/T+1j/qNzGnnROlofM2kx6u5bYMRZrjHQL5te3GGSkUzK7p3KL0WZaKdIzmA2RfnHnzp1Yl2IN2ZyaDozs3uIdUV2LE2V6Fmqf6fvIXkyL0pKfzjQ/pHMym6Vce5Trrcr1OmQvlsOP1p+tPdLJWD/MLsjuTWtDttwSp8/2SGsbxWo0XSS1zeKukT6wivcL239J12r9I72V2aZpK2mfNA0Wldu70/ZUsi3L2Ut231Vas5xghRBCCCF0TD6wQgghhBA6Jh9YIYQQQggdM/Q4WC/VNZgug/yvFhuJ4swYlofKfN8E6TJWrlyJdXfs2IHlLXmhSHdkdQ3S/JhugTQDptXYtm3bQJnFiTJIw2G+eponywFn/SNaNA6mZyI9g2l4TAtI/bY8iaThsLVgMWxIQ2VjsWfPnoEy07NQ7kvLRWgxk0gTYXo0Gntrm+0BVN9+j3RVLfGOKFZVlecopLEw2yItUau2h3SYphskmzU7JI2grXXSSlbxfmGaNpoT28ta9FoWP4z2IovnRL9nejTLR0pjZPshxWkzeyO7t33IdJG0T1r8MBojswvTv9EYteQp7oqcYIUQQgghdEw+sEIIIYQQOmaoLsLTTz+97wjbjkTJNWPHtXYdm47Q7aiVnmHXQS1FBR3B2pEoHQ/blVkKQ2HHspaug7CjZOqfXf9fu3btQJm55uw6L/Wv5aqxXR+3lApUPjY2dsK/Z8+lcTN3lLl4qdzskI7QzT1ra4f6YlebyZYtzQmV2xib64naYe4WcovZtfIW15O5g8llZ2ud9hZb6yaDoL6Y+5LqmvvS1irNiY0F2aHtF7bfE7avk7vTxo3GyNKlmeuQ1o6NG71fKMRGFackszbYfrh8+fKBshaXpO0XNNf27rT9kNa77XvUb3MRWptJamCp6ujbwNZTKznBmgJaPmxCCCFMf+wPkxCMfGCFEEIIIXRMPrBCCCGEEDpmVs8cvVPI1q1ba/369TUyMtLn/zT9DGEpUey6MmG+c7quanXN30+6DLquXsV+a/MXk27B9BBWTroD0waQeZhmhMbIdEd09buK58+0GlTXjvHtCjLpQ2yu6fdMD0H9pjAWVa6ho1AGloaJtDZmmzbXhOl1LOQIQWNh2gnTjNAzrB9kW3at3LQdpOGwurQeTPNFc2rX/M0uKA2PjSfp0exqu70K6NnWZtqXTZtH82ThO0yzSevPxq0lrQ6tHWuDjQWFvbB9nbSVZkPWP2qfaQHJhkznRs+1kB42FrR2bD3Ru5PSDlV56AXTVhKkMbR3zsTExAk/tyonWFNCV3mMQgghhDAzyQdWCCGEEELH5AMrhBBCCKFjhhoHq9fr9fn9TT+zaNGigTLTM7XoZ8w/TboT0zhQPBHTOJjPmbQILTF3LL3I+eefj+Wkn7GYJBTvxGJmkZ7J4omYPogw7QPZANlKlWs7SPNjOgnSF5jmj+J5WYwY0slUsb6EnlvVpluwdUbhRSztCPW7ZdxsPmg9VXGbbU1SP8yGTM9CGgxbvzR/pkWheF42/6YDoXhzpjkh7Zppe8wuaH+xvYXG2catJXWNtY32WguTQ/u9jQXtLS2aoSoeN9M50f5r+k7bc2it2nvE1t+JPtf00tY/2mft3UB2YWvE9rjFixcPlJlt0TeDaXZbyQlWCCGEEELH5AMrhBBCCKFj8oEVQgghhNAxQ9VgHT16tM//2ZIryOqaD5h8xqa/IN1RSz68X//611jXtEvnnXfeQJnFOyKtxdKlS7Guxeyg+qZRIf805f+q4jE2DYBpvkjPYHVJ09SiZ6ri/pnOiWK/mB6CxsI0WDt37jzh37NntOTesrhEpPmxujRPFh+NMK2V2SFpOGjdVLE2y9aTaYlI82P9o9+znIpky2bftl/QXNtYUD9MSzQ+Po7lFDfP2twSY410QKZ9MXshzaXtObQmLZ4TlZtey7RyFNvK+kHrzLRkpl2i8TQNJT2jZf3a3mLvPrJD28vo/dLy7qxqiwlGe7jFzWslJ1ghhBBCCB2TD6wQQgghhI4Zqovw5Ueulv5m3759A2Ut17mr2KWxZ88erEtuJrvWauEiyP1IrkcrtyNKOsY1l521mdyo9gxyodi1ZHPP0pxMTk5iXTrStiN4sgG7imttpiNtG3saI7LNqqpVq1ad0G9VVa1cuRLLybYsDAm5IyxciLnG6RnmCqJxtv5ROgtbvxTqo4r7bWNBNmRu9JbwDeb+oivhu3btwrpkF9YGc8+Su8RcoBQW4pJLLsG6tkbIHWTuMmqbudFp32vdyyjEjaVbor2MxqeK7XDu3LnoZjTXGrlWrX+0nkyqYPshPdtc1WQvFuLG3HO0/swO6Z1qsg3aG8yVb6m8aJ5of6til7K5VlvJCdYUYC+wkxH74A0hhJlES67c1zv2x03oJx9YIYQQQggdkw+sEEIIIYSOGaoG69RTT+3zz5tPtuXqvkHPML8u6QssLYc9g/piuhMK9WAaJdKzWGgC8/dTGg9Lt0M+fNM4kKbCjpKXL19+wm2z67x01dj0Ana835Kyh663mx1SOA2zIXsGuVdNY0Z6q9ZwIdQ+06iQpsk0eHT932zW7ILm2tJkkKbNbNZc2NQXuzZPY2R2RevBNFimXaL6Zls0ni3pj6p4HzF9EPXbtGu0J5sGz/ZOuqZv1/FpTzbdEa0d0zRayALCtGtkb2azNtcUWsLWWYuMhcbeUkEZNHZmb6SVsjViITlojMzuyQZavy+MnGBNAfahGEIIIYSTA/zA+rM/+7NatmxZzZo1q+69995j5Q8++GBt3LixRkdHa926dbV58+Zj//bkk0/WRz7ykVq1alWtXr267rjjjqlvfQghhBDCNAQ/sK6++uravHnzgBvgpptuqiuuuKJ27txZX/va1+raa689duT5+c9/vk477bTatWtXbdq0qW688Ua9shlCCCGE8HoGHY1vf/vbsfLtt99+TE+xYcOGGhkZqc2bN9dVV11V3/zmN+sf/uEfquoFv//b3/72uvPOO+sP//AP9cfPOOOMPl1DazoLwrQB5Pu2uhTXxHzAFJPEYu5YShSKpbRs2TKsS9oA64dpA6h/FJ+nirVEFpNm//79WE5Yugfy1VvcF9IBUEyiKo+xRhoc066RX960DBSfxfz6FjOJ9CiWSoTGyNpmWiLSPlhsJNKCWdoKKrf4YWYXpKkgPWIVxw8zbY/ZLPXPtGskCbD9gp5rujqD4mOZXof0M9YPiylEa8Ri+tHeYnXJBqxtFptubGxsoMy0UlRumijaZ036YbpB2petfxST0fZZS2lEfbF9z94ZBL1HTG9pmi9avxTDrIptmWKKVfk6I0zLS22z32vlhDVYhw4dqqNHj/Y1ctmyZccMY8+ePX0fFi/9txBCCCGEk4kmkfvL/0p++V9eL/331r/KQgghhBBeL5zwXcQXjxofeuihY6dYk5OTx47elyxZUhMTE33/9r73vQ+f9aIr4uVHx3al367oEpa2gI5P7Yo2tcOOkokW94CVW5+pf9YPO5qn+tbmV1vXsLGgNtuxOj3D+twyf2aHVG72RmPUknamim3A2kYuQuuzzR+Nnc0pta2LMbb5o2P8lnlqtVma15b1a22j507lvke02kXLfvhqXaBdjIW1zWyLaOlHF3s19duea+2gZ7e0zaBn2P/f5BxUbs9oGYuW/d5+j/YWm6etW7fW2rVr1UU6QO84LF26tPeLX/zi2P++4YYbep/61Kd6vV6vd8899/QWL17cO3LkSK/X6/U+9alP9W644YZer9frjY2N9ebPn987dOgQPvef/umfelWV//Jf/st/+S//5b/8N2P++8lPfnK8z6Y+ZvXgc/hjH/tY3XnnnXXgwIG64IIL6uyzz65du3bVwYMH6/rrr6/x8fGaO3duffnLX653vOMdVfWCMO2jH/1o/eQnP6nZs2fXZz/72br66qtf/uiqekHctmnTplq2bJkmNA0hhBBCmE60nGDhB1YIIYQQQnjlJJJ7CCGEEELH5AMrhBBCCKFj8oEVBnj66afrAx/4QK1evbre/OY318aNG2tiYqKqjp8uKUwvPv3pT/elu8rczQyeeeaZ+vjHP16jo6P1xje+sa677rqqyvzNFDZt2lTr16+vt7zlLbVu3br6x3/8x6rK/E1Hpjwt4AnL4cNJw1NPPdX79re/3Tt69Giv1+v1/vZv/7b37ne/u9fr9Xp/9Ed/1HeTdMmSJcdukobpw09+8pPexo0be0uWLDl2EzhzNzP4i7/4i96f/umfHlt/+/fv7/V6mb+ZwNGjR3vnn39+72c/+1mv1+v1xsfHe6eddlrv0UcfzfxNQ+6+++7e3r17ByImHG+uPv3pT/dFTBgZGek9/PDD+Px8YIX/ky1btvRWrlzZ6/V6vbPOOqv34IMPHvu3DRs29L7//e8PqWWBePrpp3tXXHFFb2xsrG/jyNxNfx5//PHeueee23vssccG/i3zN/158QPr7rvv7vV6vd7Pfvaz3oIFC3rPPPNM5m8a8/IPrOPN1aWXXtq75557jv3bNddc0/va176Gzz3hQKPh5OWLX/xivf/97/8/0yWF6cEnP/nJuu666/pypGXuZga7d++uefPm1Wc+85n67ne/W2eccUbdfPPN9eY3vznzNwOYNWtW3X777fXBD36wzjrrrDp8+HDdcccd9dhjj2X+ZghdpgWMBiscl89+9rO1c+fO+pu/+Zuq+r/TJYXh8sMf/rC2bNlSN95448C/Ze6mP0eOHKmxsbG69NJL68c//nF96Utfqg9/+MP13HPPZf5mAM8991x97nOfqzvvvLMmJyfre9/7Xt1www1VlfU3k+gqLWA+sILy+c9/vu644476t3/7tzrzzDP70iW9yEvTJYXhc/fdd9e2bdtq+fLltWzZstq3b1+95z3vqXvuuaeqMnfTnaVLl9bs2bPr2muvraqqyy+/vJYvX1733XdfVWX+pjs//elPa//+/fW2t72tqqo2bNhQCxYsqJ///OdVlfmbCfxf77kX0wLSv72cfGAF5Atf+EJ94xvfqO985zt13nnnHSu/5ppr6tZbb62qqi1bttSBAwfqyiuvHFIrw8u56aabav/+/TUxMVETExO1aNGi2rRpU733ve/N3M0ALrjggnrXu95VmzZtqqoXNu/x8fFas2ZN5m8GsHjx4tq3b19t3769qqp27dpVu3fvrtWrV2f+ZhDHm6uX/tv4+Hjdfffd9Xu/93v8oI61YuF1wN69e3tV1VuxYkXv8ssv711++eW93/zN3+z1er3egQMHeu9+97t7q1at6l166aW9H/zgB0NubTgeLxVvZu5mBrt37+694x3v6K1bt653+eWX9+64445er5f5myl8/etf761bt6532WWX9d70pjf1vvGNb/R6vczfdOTGG2/sLVy4sDdnzpzeyMjIsctcx5urxx9/vPehD32ot3Llyt7o6GjvX/7lX/T5SZUTQgghhNAxcRGGEEIIIXRMPrBCCCGEEDomH1ghhBBCCB2TD6wQQgghhI7JB1YIIYQQQsf8f7TuaQstP4odAAAAAElFTkSuQmCC"
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heatmap(xData[:,:,1,3]',yflip=true,legend=false,color=:greys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13-element Array{Float64,1}:\n",
       "  1.0\n",
       " 67.0\n",
       " 43.0\n",
       "  0.0\n",
       "  0.0\n",
       "  0.0\n",
       "  0.0\n",
       "  1.0\n",
       "  0.0\n",
       "  0.0\n",
       "  0.0\n",
       "  0.0\n",
       "  0.0"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yData[:,3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Chain(Conv((3, 3), 1=>8, relu), BatchNorm(8), #96, Conv((3, 3), 8=>4, relu), BatchNorm(4), #97, flatten, Dense(2116, 80, relu), #98)"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Flux\n",
    "\n",
    "function buildLocalizationModel()\n",
    "    Chain(  #Assuming 100x100\n",
    "      Conv((3, 3), 1 => 8, relu, pad=(0, 0), stride=(1, 1)),   \n",
    "      BatchNorm(8),\n",
    "      x -> maxpool(x, (2, 2)),  #16x16x8\n",
    "      Conv((3, 3), 8 => 4, relu, pad=(0, 0), stride=(1, 1)), #14x14x4  convolutional layer\n",
    "      BatchNorm(4),\n",
    "      x -> maxpool(x, (2, 2)), #7x7x4 ,\n",
    "      flatten,  #196 neurons\n",
    "      Dense(2116, 80, relu),\n",
    "      (prev)->vcat( sigmoid.(Dense(80,1)(prev)),\n",
    "                    Dense(80,1)(prev), #x \n",
    "                    Dense(80,1)(prev), #y\n",
    "                    softmax(Dense(80,10)(prev))...)\n",
    "    )\n",
    "end\n",
    "\n",
    "modelLocalization = buildLocalizationModel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13×1 Array{Float32,2}:\n",
       "  0.495603\n",
       " -0.020295162\n",
       " -0.0145284915\n",
       "  0.096858144\n",
       "  0.10102488\n",
       "  0.102372065\n",
       "  0.102341905\n",
       "  0.10374058\n",
       "  0.09806526\n",
       "  0.09576972\n",
       "  0.10160684\n",
       "  0.09938787\n",
       "  0.09883269"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampleImage = xData[:,:,:,[10]]\n",
    "# debugModel(modelLocalization)\n",
    "modelLocalization(sampleImage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Extra task:** Finish off the simple localization example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.5.3",
   "language": "julia",
   "name": "julia-1.5"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
